LSI (Avago) 9207-8i with Seagate 10TB Enterprise (ST10000NM0016)

Chris Moore · Jan 10, 2018

brianm said:
I personally feel that the controller is a weak link in the FreeNas concept, it and its associated cables are a fault waiting to happen but until we get a greatly increased number of on-board SATA connectors we are stuck with controllers.

For massive drive counts, you are always going to have some sort of SAS controller or something that is cobbled together in such a way that it is not desirable, like the HighPoint Rocket 750 40-Channel SATA. SATA was never intended for a high drive count system like a server.

kmr99 · Jan 10, 2018

Brian, thanks for your thoughts. One had been operating correctly for 3 months, but as I wrote in data point "2." above, after querying the controller information yesterday on the system that had been working properly with sas3flash, about 30 minutes later I began getting SYNC CACHE timeout errors on 5 of the 8 drivers and 3 drives faulted within 5 hours. So, at this point, I have two systems where drives are being faulted and I am at risk of losing my zpool. As I wrote above, I've already switched controllers, cables, and backplanes.

I am not blaming FreeBSD. From what I've found with Google, these errors (FLUSHCACHE48 with ACHI drivers) and SYNCHRONIZE CACHE(10) with 3008 drivers have been limited to half a dozen different people with Seagate 10GB drives with problems persisting despite changing cables and other hardware. I suspect the issue is with the drive's firmware. One poster mentioned his problems resolved with a change to Linux and it's mptsas drivers. Since it is not feasible for me to replace the 16 Seagate drives, I need to find an OS/driver combination that is compatible.

skyyxy · Jan 10, 2018

miip said:
I tried contacting seagate about the issue, but they only told me to check the drive with their tool and report back. Maybe if more users would contact them they would start investigating this.

I also thought about getting a newer 3008 series controller.

3008 is same, I tested. my box is ST enterprise HDDs 6 or 8TB with LSI 24slots 4U case with LSI (9300-8i) 3008 chipset adapter. The 4U case has LSI SAS12Gb extend chipset onboard and if I connect the 9300-8i that will going like your issue(9211-8i is same),but if I reinstall the system to 9.10 leastest will be ok, so I think its about the FNS system or FreeBSD system. Hope can be work at later version(I not try on 11.1 yet).

skyyxy · Jan 10, 2018

miip said:
This could also be a simple driver issue. LSI 2008 and 2308 use the same driver in FreeBSD.

So far no errors with the LSI 3008 controller.

BTW, where did you guys see that LSI 2008 only supports drives <8TB? I can not find anything official about a 8TB limit. There were no issues for several weeks, so i don't think this is some compatibility issue.

no limit atleast to 10TB in my box (ST 10TB enterprise HDD *24)

kmr99 · Jan 10, 2018

skyxy, when you say 3008 is the same, are you referring to SYNC CACHE timeout errors? Are you getting those with your 24 Seagate drives? You mentioned if you reinstall version 9.10, things are ok. What sort of problems did you notice with a different version?

skyyxy · Jan 10, 2018

kmr99 said:
skyxy, when you say 3008 is the same, are you referring to SYNC CACHE timeout errors? Are you getting those with your 24 Seagate drives? You mentioned if you reinstall version 9.10, things are ok. What sort of problems did you notice with a different version?

Totally same like the picture in the topic.
Still can boot on and got all 24 dirvers but you need wait a long time(over than 1 hour)
Use lsi 3008 or 2008 adapter with FNS 9.10 lastest is working atleast for me.
Use lsi 3008 or 2008 adapter both not working on FNS 11.0 or .1 atleast for me
My case is the 24docks 4U with 1 LSI 12Gb extend chipset in side so I think may be is about the extend chipset, because never happen with LSI 6Gb extend chipset case (I tested).
My english is not good, so hope you can understand.

Chris Moore · Jan 10, 2018

Does this sounds like a BSD bug?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

kmr99 · Jan 10, 2018

Chris Moore said:
Does this sounds like a BSD bug?

I don't think so, given I've gotten FLUSHCACHE48 errors on all 6 drives in the backup server with both the X10SDV motherboard SATA ports and ACHI drivers as well as now SYNCHRONIZE_CACHE(10) errors with the LSI 9300 and SCSI drivers. Since the error occurs with both drivers, I wouldn't consider this a BSD bug. Neither, do I think it is a signal integrity problem as the drives are not showing any significant number of CRC errors. I am still suspicious of an incompatibility with the BSD drivers and the Seagate firmware. Looking at /usr/src/sys/cam/scsi/scsi_da.c, there is a table of a few dozen SCSI device quirks and maybe adding an appropriate quirk entry for the drives might mitigate the issue. For the ACHI driver, it appears that /usr/src/sys/cam/ata/ata_da.c has a similar table of quirks. On a brief review, the source for the MPR driver (/usr/src/sys/dev/mpr) is organized quite a bit differently than the Linux mpt3sas driver.

Evi Vanoost · Jan 10, 2018

You're using SATA drives on SAS controllers, does the system actually recognize the drives as SATA drives? Can you pull the smartctl? Any issues in the SMART?

The problem I've noticed with SAS expanders is that if one SATA drive goes, the rest may get pulled down with it especially given that you have a timeout, most likely the single bus it has is "used" by a single bad or lagging drive. Check your SAS expander logs, they sometimes contain more info about which drive is using up the bus.

Do you have a wonky SSD drive perhaps, that's a more likely source of firmware issues but I would pull the drives one by one and do stress-tests.

From one blog post:
Aug 30, 2010: Update: At a significant account, I can say that we (meaning Nexenta) have verified that SAS/SATA expanders combined with high loads of ZFS activity have proven conclusively to be highly toxic. So, if you're designing an enterprise storage solution, please consider using SAS all the way to the disk drives, and just skip those cheaper SATA options. You may think SATA looks like a bargain, but when your array goes offline during ZFS scrub or resilver operations because the expander is choking on cache sync commands, you'll really wish you had spent the extra cash up front. Really.

So something about cache sync commands timing out...

kmr99 · Jan 11, 2018

Evi Vanoost said:
You're using SATA drives on SAS controllers, does the system actually recognize the drives as SATA drives? Can you pull the smartctl? Any issues in the SMART?

Thanks for your thoughts, Evi. No, there are no issues with SMART nor does there appear to be any errors with the data on the disks being faulted. I'll attach the smart output for one device below. You have a good memory associating the expanders with cache sync commands! However, I'm not using expanders and the issue occurs both with drives directly attached as well as attached through backplanes.

Code:

root@freenas1:~ # smartctl -a /dev/da7
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Seagate IronWolf
Device Model:	 ST10000VN0004-1ZD101
Serial Number:	ZA218PW9
LU WWN Device Id: 5 000c50 0a2589bee
Firmware Version: SC60
User Capacity:	10,000,831,348,736 bytes [10.0 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	7200 rpm
Form Factor:	  3.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Thu Jan 11 13:22:51 2018 MST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
										was completed without error.
										Auto Offline Data Collection: Enabled.
Self-test execution status:	  (   0) The previous self-test routine completed
										without error or no self-test has ever
										been run.
Total time to complete Offline
data collection:				(  567) seconds.
Offline data collection
capabilities:					(0x7b) SMART execute Offline immediate.
										Auto Offline data collection on/off support.
										Suspend Offline collection upon new
										command.
										Offline surface scan supported.
										Self-test supported.
										Conveyance Self-test supported.
										Selective Self-test supported.
SMART capabilities:			(0x0003) Saves SMART data before entering
										power-saving mode.
										Supports SMART auto save timer.
Error logging capability:		(0x01) Error logging supported.
										General Purpose Logging supported.
Short self-test routine
recommended polling time:		(   1) minutes.
Extended self-test routine
recommended polling time:		( 849) minutes.
Conveyance self-test routine
recommended polling time:		(   2) minutes.
SCT capabilities:			  (0x50bd) SCT Status supported.
										SCT Error Recovery Control supported.
										SCT Feature Control supported.
										SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000f   083   064   044	Pre-fail  Always	   -	   222338832
  3 Spin_Up_Time			0x0003   088   086   000	Pre-fail  Always	   -	   0
  4 Start_Stop_Count		0x0032   100   100   020	Old_age   Always	   -	   56
  5 Reallocated_Sector_Ct   0x0033   100   100   010	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000f   091   060   045	Pre-fail  Always	   -	   1181286135
  9 Power_On_Hours		  0x0032   095   095   000	Old_age   Always	   -	   4730 (35 13 0)
 10 Spin_Retry_Count		0x0013   100   100   097	Pre-fail  Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   020	Old_age   Always	   -	   36
184 End-to-End_Error		0x0032   100   100   099	Old_age   Always	   -	   0
187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
188 Command_Timeout		 0x0032   100   095   000	Old_age   Always	   -	   17180131343
189 High_Fly_Writes		 0x003a   073   073   000	Old_age   Always	   -	   27
190 Airflow_Temperature_Cel 0x0022   063   056   040	Old_age   Always	   -	   37 (Min/Max 32/40)
191 G-Sense_Error_Rate	  0x0032   098   098   000	Old_age   Always	   -	   5344
192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   20
193 Load_Cycle_Count		0x0032   100   100   000	Old_age   Always	   -	   548
194 Temperature_Celsius	 0x0022   037   044   000	Old_age   Always	   -	   37 (0 23 0 0 0)
195 Hardware_ECC_Recovered  0x001a   008   001   000	Old_age   Always	   -	   222338832
197 Current_Pending_Sector  0x0012   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0010   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x003e   200   200   000	Old_age   Always	   -	   18
200 Multi_Zone_Error_Rate   0x0023   100   100   001	Pre-fail  Always	   -	   0
240 Head_Flying_Hours	   0x0000   100   253   000	Old_age   Offline	  -	   4432 (76 157 0)
241 Total_LBAs_Written	  0x0000   100   253   000	Old_age   Offline	  -	   89324756190
242 Total_LBAs_Read		 0x0000   100   253   000	Old_age   Offline	  -	   85307597514

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	   726		 -
# 2  Extended offline	Aborted by host			   90%	   707		 -
# 3  Extended offline	Interrupted (host reset)	  60%	   706		 -
# 4  Short offline	   Completed without error	   00%	   701		 -
# 5  Extended offline	Completed without error	   00%	   583		 -
# 6  Short offline	   Completed without error	   00%	   568		 -
# 7  Extended offline	Completed without error	   00%	   513		 -
# 8  Short offline	   Completed without error	   00%	   498		 -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

cobrakiller58 · Jan 11, 2018

That command timeout count sent a chill down my spine and 199 seems high to me. Are the crc errors all above zero on your drives or was this one abnormal?

kmr99 · Jan 11, 2018

cobrakiller58 said:
That command timeout count sent a chill down my spine and 199 seems high to me. Are the crc errors all above zero on your drives or was this one abnormal?

No, most of the CRC error counts are 0. Drive da3 initially had many CRC errors, but the count has been steady for months after replacing the SATA cable. The output of my monitoring script is below.

Thanks for looking at the SMART output - I'm not very experienced interpreting the data. For the command timeout, I see the high raw number. I find it hard to believe there have been 17 billion timeouts. For some fields,I thought the raw number was a vendor specific value and one should look at the VALUE and WORST fields to interpret the data.

Code:

da2 32 ZA224RKK ST10000VN0004-1ZD101 FW:SC60 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/ 64983316  CrcErrs:  0 POH:250
da3 34 ZA2185XW ST10000NE0004-1ZF101 FW:EN01 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1166474142 CrcErrs:828 POH:4670
da4 36 ZA21420W ST10000VN0004-1ZD101 FW:SC60 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1189435465 CrcErrs:  8 POH:4734
da5 32 ZA21ECB7 ST10000VN0004-1ZD101 FW:SC60 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1201062164 CrcErrs:  0 POH:4756
da6 34 ZA218TQ9 ST10000NE0004-1ZF101 FW:EN01 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1183631604 CrcErrs:  0 POH:4670
da7 38 ZA218PW9 ST10000VN0004-1ZD101 FW:SC60 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1181872948 CrcErrs: 18 POH:4732

kmr99 · Jan 11, 2018

kmr99 said:

I find it hard to believe there have been 17 billion timeouts.

Code:


Wow, it seems like there has been 17G timeouts for that one drive. I added timeout to my monitoring script and below is the info for my two pools. I'll monitor the timeout count for that drive and see if it continues to increase

Code:
Backup pool
da2 32 ZA224Rxx ST10000VN0004-1ZD101 FW:SC60 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/  65082246 CrcErrs:  0 TimeOut:		  0 POH: 251
da3 34 ZA2185xx ST10000NE0004-1ZF101 FW:EN01 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1166568869 CrcErrs:828 TimeOut:		741 POH:4670
da4 36 ZA2142xx ST10000VN0004-1ZD101 FW:SC60 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1189532952 CrcErrs:  8 TimeOut:		  4 POH:4735
da5 33 ZA21ECxx ST10000VN0004-1ZD101 FW:SC60 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1201095943 CrcErrs:  0 TimeOut:		 52 POH:4757
da6 34 ZA218Txx ST10000NE0004-1ZF101 FW:EN01 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1183729909 CrcErrs:  0 TimeOut:		  0 POH:4670
da7 38 ZA218Pxx ST10000VN0004-1ZD101 FW:SC60 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1181970564 CrcErrs: 18 TimeOut:17180131343 POH:4732

Main pool
da0 32 ZA20JBxx ST10000NM0016-1TT101 FW:SND0 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1106833440 CrcErrs:  0 TimeOut:		  0 POH:4707
da1 33 ZA2248xx ST10000VN0004-1ZD101 FW:SC60 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/ 154099372 CrcErrs:  0 TimeOut:		  1 POH: 668
da2 31 ZA20SRxx ST10000NM0016-1TT101 FW:SNB0 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1133367137 CrcErrs:  4 TimeOut:		  0 POH:4802
da3 28 ZA2129xx ST10000NE0004-1ZF101 FW:EN01 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1082197763 CrcErrs:  0 TimeOut:		  4 POH:4318
da4 27 ZA20MPxx ST10000NM0016-1TT101 FW:SNB0 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1285734573 CrcErrs:  0 TimeOut:		  0 POH:4828
da5 25 ZA21E4xx ST10000VN0004-1ZD101 FW:SC60 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1101085143 CrcErrs:  0 TimeOut:		 10 POH:4758
da6 24 ZA21DYxx ST10000VN0004-1ZD101 FW:SC60 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1098922562 CrcErrs: 11 TimeOut:		  2 POH:4736
da7 27 ZA20BWxx ST10000NE0004-1ZF101 FW:EN01 ReAlloc:0 RepUnc:0 CurPend:0 OffUnc:0 SeekErrs:0/1087043042 CrcErrs:  0 TimeOut:		  1 POH:4672


For those interested, below is the simple awk script that I use:

Code:
#!/bin/sh
for i in $(sysctl -n kern.disks | awk '{for (i=NF; i!=0 ; i--) print $i }' )
do
  (echo "DeviceLabel: $i" ; smartctl -v 7,hex48 -a /dev/$i) | awk '\
  /DeviceLabel:/{DevLabel=$2;} \
  /Temperature_Celsius/{DevTemp=$10;} \
  /Serial Number:/{DevSerNum=$3;} \
  /Seek_Error_Rate/{SER1=("0x" substr($10,3,4));SER2=("0x" substr($10,7))}; \
  /Device Model:/{DevName=$3} \
  /Power_On_Hours/{POH=$10} \
  /Firmware Version:/{FIRMWARE=$3} \
  /Reallocated_Sector/{ReAlloc=$10} \
  /Reported_Uncorrect/{RepUnc=$10} \
  /Current_Pending_Sector/{Pending=$10} \
  /Offline_Uncorrectable/{OffUnc=$10} \
  /UDMA_CRC_Error_Count/{CrcErr=$10} \
  /Command_Timeout/{TO=$10} \
  END { if (DevTemp!="") printf "%s %s %s %s FW:%s ReAlloc:%d RepUnc:%d CurPend:%d OffUnc:%d SeekErrs:%d/%10d CrcErrs:%3d TimeOut:%11d POH:%4s\n", \
DevLabel,DevTemp,DevSerNum,DevName,FIRMWARE,ReAlloc,RepUnc,Pending,OffUnc,SER1,SER2,CrcErr,TO,POH; }'
done

Johnnie Black · Jan 12, 2018

kmr99 said:
For the command timeout, I see the high raw number. I find it hard to believe there have been 17 billion timeouts. For some fields,I thought the raw number was a vendor specific value and one should look at the VALUE and WORST fields to interpret the data.

For those disks it's a multi byte number, RAW value is meaningless if read as a single number.

cobrakiller58 · Jan 12, 2018

Johnnie Black said:
For those disks it's a multi byte number, RAW value is meaningless if read as a single number.

I'll have to keep that in mind when comparing my 8TB to the newer 10+TB drives

micahtangelo · Jan 13, 2018

We're experiencing these issues too. We use lots of 10TB Seagate drives (MSP in Media/Entertainment sector). Other searching in the forums on these error buzzwords pointed to some possible issues with the controller firmware, but this thread has really illuminated that it's probably a problem centered around the drives themselves. I'm installing CentOS as I type this to see if we get anything like this by changing only the OS.

kmr99 · Jan 13, 2018

micahtangelo said:
We're experiencing these issues too. We use lots of 10TB Seagate drives (MSP in Media/Entertainment sector). Other searching in the forums on these error buzzwords pointed to some possible issues with the controller firmware, but this thread has really illuminated that it's probably a problem centered around the drives themselves. I'm installing CentOS as I type this to see if we get anything like this by changing only the OS.

I'm leaning towards a drive issue as well given issues across several controllers. In this thread, though, there are still a few things that suggest that it might not be a pure drive issue. Given the poster who has not had issues with 24 of these drives with a Debian-based Linux, I'm interested to hear about your experience testing with CentOS.

wblock · Jan 13, 2018

Has anyone entered a bug for this? If it is a problem with the drives, it might be possible to work around them in the driver. As, in fact, it sounds like Linux has done.

micahtangelo · Jan 18, 2018

Was able to import the pool OK in CentOS, but getting a very large amount of (different) errors, and the disks are still being kicked out of the pool eventually (quite quickly) due to too many errors. We're in a very bad position here because, and please be patient with us, we had this pool in a striped configuration (intended for a *very* short time) and have lost data. They were brand-new disks, used fairly intensively for about a week, and now 3 of the 4 drives are spitting errors like they're very broken. I don't have time to pull logs right *now* but will get to it asap.

Since the errors look different it may be that some incompatibility between the drives and FreeNAS driver caused permanent issues on the drives. But that's a flailing WAG. Just posting here to signal my intention to return with more useful information - please let me know what information would be most useful; I have both CentOS 7.4 and FreeNAS 11.1 installed on the machine, so can boot into either to get whatever you want.

kmr99 · Jan 26, 2018

micahtangelo said:
Was able to import the pool OK in CentOS, but getting a very large amount of (different) errors, and the disks are still being kicked out of the pool eventually (quite quickly) due to too many errors.

Like you, I was able to import my pool with Linux (Ubuntu 17.10). I continued getting errors (I/O errors) on the drives, but none that affected the pool.

Since the errors look different it may be that some incompatibility between the drives and FreeNAS driver caused permanent issues on the drives.

While I haven't considered drives to be permanently affected by drivers, there is one behavior of my drives that makes me wonder if something similar could be the case. Specifically, I have one server with 8 of these drives that had no issues for 3 months. Then, within 30 minutes of running the sas3flash utility to inquire the firmware version of the LSI 3008 (9300-8i) card, I started getting the SYNCHRONIZE CACHE timeout errors on all the drives in the pool and those intermittent timeout errors have continued since -- bizarre behavior that I can't explain

Important Announcement for The TrueNAS Community.

LSI (Avago) 9207-8i with Seagate 10TB Enterprise (ST10000NM0016)

Hall of Famer

Dabbler

Contributor

Contributor

Dabbler

Contributor

Hall of Famer

Dabbler

Explorer

Dabbler

Guru

Dabbler

Dabbler

Guru

Guru

Dabbler

Dabbler

Documentation Engineer

Dabbler

Dabbler

Similar threads

Important Announcement for The TrueNAS Community.