Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

LSI2008 (M1015 cross-flash) disk detection issues

Joined
Oct 24, 2016
Messages
10
Hi everyone,

I've been pounding my head against the desk since Friday, and before I break my desk/kb/head any further, I thought I would post here and see if anyone could provide some guidance. Before getting to that, I should mention I am a FreeNAS n00000000b but can generally find my way through tasks/projects I don't fully understand - all this to say, please forgive my ignorance and if I'm doing something stupid, please don't hold back calling me out on it. Also, apologies for the book below, I'll try to be thorough as I won't be able to check in/answer questions for a day or two and would like to provide ALL relevant info.

Problem: LSI2008 only detects 2 of 8 disks, the symptoms are specific to the 6 disks meaning it doesn't matter which port/bay the drives are plugged into the same 6 disks always fail, and the 2 working ones always work. The onboard controller detects all 8 (when I plug them there instead of LSI, obviously). My best guess is theres a compatibility issue between the Hitachi firmware & LSI2008 P20 - I've never heard of this type of compatibility, but everything I've done is leading me to believe its the case.

Build:
FreeNAS 9.2.latest & FreeNAS 9.3.1
Board: Supermicro A1Sai-2750f
CPU: Intel® Atom processor C2750
RAM: 2x8GB ECC
RAID controller: IBM M1015, crossflashed to LSI2008 P20
Disks: 7x Hitachi MUA721010kLA330 1x Seagate rando
Case: Silverstone DS380 (8x 3.5 HS bays and some proprietary "backplane" with 2x molex, 8x SAS or 8x SATA)

Background:
The entire build above, minus the disks/LSI2008 were in working order last year but I shut the server down and didnt use it for some time. When I got my new (but used/old) drives/card, I decided to spend the time to resurrect it. Using the original IBM FW, all drives were detected and I did a test ZFS pool, all drives in working order. I did not want the M1015 card to get in the way of ZFS (as recommedned on this forum) so I cross-flashed with LSI2008 P20 from Avagos site - what a bloody nightmare, sas2flash would work cause "non-supported" card (yes im an idiot, I shouldve wiped fw first) then MegaRec wouldnt run on this system (known Intel compat issue). So, installed the card in another system, wiped the FW & upgraded to P20 w/ the bios support (short-term for testing purposes). Once reinstalled in my system, FreeNAS nor the LSI BIOS are able to detect 6 of the drives regardless of how many drives are installed or where they're plugged into the LSI card - again, onboard SATA detects all drives.

Why I think its FW compatibility issue:
As shown in the list below, the 2 working drives are on FW GKAOAB0A and the 5 "dead" drives are on GKAOA70M.

Model,SerialNumber,FirmwareVersion,TestIntel,TestLSI (ports 4,5,6)
Hitachi HUA721010KLA330,GTA060PBK1AYRF,GKAOA70M,YES,NO
Hitachi HUA721010KLA330,GTE002PBJ52EUE,GKAOAB0A,YES,YES
Hitachi HUA721010KLA330,GTA060PBK14ZMF,GKAOA70M,YES,NO
SEAGATE ST31000340NS,9QJ2W7LH,XR1A,YES,NO
Hitachi HUA721010KLA330,GTA060PBJSH8TF,GKAOAB0A, YES,YES
Hitachi HUA721010KLA330,GTA060PBK1A8YF,GKAOA70M,YES,NO
Hitachi HUA721010KLA330,GTA160PBKD8UWF,GKAOA70M,YES,NO
Hitachi HUA721010KLA330,GTA160PBKBYVMF,GKAOA70M,YES,NO

Other notes:
All drives have been plugged into onboard and booted into a Linux live environment where I ran SMART tests. Although a couple do have bad sectors, they are all properly detected. I can deal with replacing the drives with bad sectors later, for now I just want to get them detected. I've tried, to no avail, to upgrade the Hitachi firmware - I can't seem to find a Hitachi/HGST utility to upgrade firmware and I've been reluctant to try the WD equivalent as these drives predate their acquisition of HGST. I did however attempt Dells nautilus util, but this also failed (logic for running Dell util is these drives are from a Dell Compellent, fiber-SATA array [had SATA2fiberChannel adapters on each drive which I removed]).

Questions:

I've found conflicting forum posts suggestion which firmware I should run some stating P16, P18 & P20 - all in IT mode obviously. Which firmware should I run if I plan to continue on FreeNAS9.3 for the foreseeable future?

Any recommendations on upgrading FW of the Hitachi POS drives?

Any general recommendations? Or steps that I can take to narrow down the cause?

How much of a n00b am I? :D ha

I'll be online for another 30 minutes or so if anyone has any questions, otherwise I will check back as soon as possible - Any assistance/guidance is greatly appreciated

Mike
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,108
I've found conflicting forum posts suggestion which firmware I should run some stating P16, P18 & P20 - all in IT mode obviously.
There is no conflicting infromation, only outdated information. P16 is for very outdated versions, P18 was never used and P20 is the current version.
Which firmware should I run if I plan to continue on FreeNAS9.3 for the foreseeable future?
The version FreeNAS tells you to run (except for SAS3).
 
Joined
Oct 24, 2016
Messages
10
ericloewe.37321 said:
The version FreeNAS tells you to run (except for SAS3).
Thank you for your response & assistance.

Can you please elaborate a little? (I'm a n00b lol). At no point during bootup do I see any errors indicating I'm on the wrong firmware for this version of FreeNAS. Furthermore, I don't see anything in the WebUI that indicates what version I'm on (or should be on) however the output below indicates I'm on P20 IT & driver version is 21. Is this what I should expect?

cat /var/run/dmesg.boot | grep mps0
Code:
mps0: <Avago Technologies (LSI) SAS2008> port 0xc000-0xc0ff mem 0xdf1c0000-0xdf1c3fff,0xdf180000-0xdf1bffff irq 22 at device 0.0 on
pci4																																
mps0: Firmware: 20.00.07.00, Driver: 21.01.00.00-fbsd																			
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>


lspci | grep LSI
Code:
04:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02)
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,862
Did the problem drives come from an existing SAN? It's possible they are locked.
 
Joined
Oct 24, 2016
Messages
10
Did the problem drives come from an existing SAN? It's possible they are locked.
Thank you for your response, yes they are from an existing SAN, but decommissioned enclosure (Dell stopped support SATA & their fiber-channel enclosures for compellent). I'm not sure if this is what you are referring to, but could be:

smartctl --xall /dev/ada0 | grep Security
Code:
ATA Security is:  Disabled, NOT FROZEN [SEC1]


Based on the above, I assume they are not locked (frozen?) - but you know what they say about assumptions....

Thanks again

Mike
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,108
the output below indicates I'm on P20 IT & driver version is 21. Is this what I should expect?
Didn't you say you were on 9.3? That output is normal for 9.10.

In any case, you're using a supported configuration.

FreeNAS gives you a warning if you're using the wrong firmware version.
 

Arwen

FreeNAS Expert
Joined
May 17, 2014
Messages
1,142
One thing I have heard happens with certain SAS controllers, is that they won't recognize SATA I disk speed, (1.5Gbps).
SAS I is 3Gbps, SAS II is 6Gbps and SAS III is 12Gbps. But, SAS is supposed to be backward compatible to SATA. Now
this is very un-likely your problem, but check to see if the affected drives are using SATA I speed, 1.5Gbps.

For example, the SSD in my laptop has this;
Code:
# smartctl -x /dev/sda | grep "SATA Version"
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,862
Can you post some pictures of what bios is showing and the output of camcontrol devlist? I'm trying to figure out what isn't showing up in FreeNAS but is showing up in other OS's.
 
Joined
Oct 24, 2016
Messages
10
Request for SATA Version
Hi Arwen, unfortunately
Code:
... | grep "SATA\ Version"
doesnt return anything as its reporting as ATA (not SATA).
Code:
 [root@freenas ~]# smartctl -x /dev/ada1 |grep ATA\ Version 			
ATA Version is:   ATA/ATAPI-7 T13/1532D revision 1							
[root@freenas ~]# smartctl -x /dev/ada0 |grep ATA\ Version					
ATA Version is:   ATA/ATAPI-7 T13/1532D revision 1	 


Note the above drivers (1 working/1 "bad") are both plugged into the onboard Intel controller (currently configured for AHCI I believe), and no the LSI2008. I'm at work at the moment, so can't make the physical rewiring changes (no access to the box) but I will when I get home this evening and will provide the results.


Can you post some pictures of what bios is showing and the output of camcontrol devlist? I'm trying to figure out what isn't showing up in FreeNAS but is showing up in other OS's.
Hi Depasseg, I can provide the output of camcontrol, but with the same limitations as above (plugged in via Intel vs LSI) - and I'll have to get back to you this evening for the LSI BIOS screenshot. Having done so before, I can provide some general information on what's displayed. The 2 drives with FW GKAOAB0A are listed in BIOS & camcontrol as connected to controller 0, port x (where x is the bay/cable the drive is connected to on the card) and the 6 other drives do not appear whatsoever in either (I will provide screenshots to backup this statement as soon as possible when I get home).

Code:
 [root@freenas ~]# camcontrol devlist   										
<Hitachi HUA721010KLA330 GKAOA70M>  at scbus1 target 0 lun 0 (ada0,pass0)	  
<Hitachi HUA721010KLA330 GKAOAB0A>  at scbus2 target 0 lun 0 (pass2,ada1)	  
<Kingston DataTraveler 3.0 PMAP>   at scbus4 target 0 lun 0 (pass1,da0)		
   
 
Joined
Oct 24, 2016
Messages
10
Hi everyone,

I've taken all the screenshots as requested, and a few others for good measure. They've been uploaded to OneDrive (I know, I know..... lets pretend I'm not using a MS service right now), I trust this is "appropriate/relevant" use of offsite links but if not and theres a general method use by this forum to share screenshots, please let me know.
Please see code snipets below, as requested.
Request for SATA Version
The above output asn't changed as the drives aren't detected. I did however cycle through several "bad" drives with the onboard controller and the output is consistent.

Code:
#   "BAD" drive | Onboard Intel controller
 # smartctl -a /dev/ada0 | tail -n +4 | head -n 15		 														
=== START OF INFORMATION SECTION ===																								
Model Family:	 Hitachi Ultrastar A7K1000																						
Device Model:	 Hitachi HUA721010KLA330																						
Serial Number:	GTA160PBKBYVMF																									
LU WWN Device Id: 5 000cca 21eefb6ae																								
Firmware Version: GKAOA70M																										
User Capacity:	1,000,204,886,016 bytes [1.00 TB]																				
Sector Size:	  512 bytes logical/physical																						
Device is:		In smartctl database [for details use: -P show]																
ATA Version is:   ATA/ATAPI-7 T13/1532D revision 1																				
Local Time is:	Tue Oct 25 16:59:16 2016 PDT																					
SMART support is: Available - device has SMART capability.																		
SMART support is: Enabled																																																				

Code:
# Good/detectable drive | Onboard Intel controller
 # smartctl -a /dev/ada1 | tail -n +4 | head -n 15 																
=== START OF INFORMATION SECTION ===																								
Model Family:	 Hitachi Ultrastar A7K1000																						
Device Model:	 Hitachi HUA721010KLA330																						
Serial Number:	GTE002PBJ52EUE																									
LU WWN Device Id: 5 000cca 21ede7e0b																								
Firmware Version: GKAOAB0A																										
User Capacity:	1,000,204,886,016 bytes [1.00 TB]																				
Sector Size:	  512 bytes logical/physical																						
Device is:		In smartctl database [for details use: -P show]																
ATA Version is:   ATA/ATAPI-7 T13/1532D revision 1																				
Local Time is:	Tue Oct 25 17:00:34 2016 PDT																					
SMART support is: Available - device has SMART capability.																		
SMART support is: Enabled																																																				  

Code:
# No other drives in sight...
# Ignoring the USB devices that are 'ad[0-9]'
# LOL @ dirty command, but whatever works
 # ls -1 /dev/ada*| grep -v '[0-9][a-z]'															
/dev/ada0																														
/dev/ada1

Code:
 # camcontrol devlist 
<Hitachi HUA721010KLA330 GKAOA70M>  at scbus1 target 0 lun 0 (ada0,pass0)															
<Hitachi HUA721010KLA330 GKAOAB0A>  at scbus2 target 0 lun 0 (ada1,pass1)															
<Kingston DataTraveler 3.0 PMAP>   at scbus4 target 0 lun 0 (pass2,da0)	


I will switch the onboard from AHCI to ATA, if any of the above tests change, I will post back.

Thanks again,

Mike

EDIT: Wow I don't think I could've messed up the syntax of code/cmd/quote anymore if I deliberately tried to do so... fixed 1,2,3,4 times
 
Last edited:
Joined
Oct 24, 2016
Messages
10
One thing I have heard happens with certain SAS controllers, is that they won't recognize SATA I disk speed, (1.5Gbps).
SAS I is 3Gbps, SAS II is 6Gbps and SAS III is 12Gbps. But, SAS is supposed to be backward compatible to SATA. Now
this is very un-likely your problem, but check to see if the affected drives are using SATA I speed, 1.5Gbps.

For example, the SSD in my laptop has this;
Code:
# smartctl -x /dev/sda | grep "SATA Version"
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Arwen you genius! I think you're onto something, I wasn't able to replicate your results but I confirmed your suspicion another way....


Booted in verbose mode, then ran the following
Code:
 #cat /var/run/dmesg.boot | egrep '(pass|da|mps)[0-9]:'  												
mps0: <Avago Technologies (LSI) SAS2008> port 0xc000-0xc0ff mem 0xdf1c0000-0xdf1c3fff,0xdf180000-0xdf1bffff irq 22 at device 0.0 on
pci4																																
mps0: Firmware: 20.00.07.00, Driver: 21.01.00.00-fbsd																			
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>								
mps0: attempting to allocate 1 MSI-X vectors (15 supported)																		
mps0: using IRQ 257 for MSI-X
 pass0: <Hitachi HUA721010KLA330 GKAOA70M> ATA-7 SATA 1.x device																	
pass0: Serial Number GTA160PBKBYVMF																								
pass0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)																			
pass1: <Hitachi HUA721010KLA330 GKAOAB0A> ATA-7 SATA 2.x device																	
pass1: Serial Number GTE002PBJ52EUE																								
pass1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)   																	
da0: <Kingston DataTraveler 3.0 PMAP> Removable Direct Access SPC-4 SCSI device													
da0: Serial Number 0019E06B9C90BE61376B7786																						
da0: 40.000MB/s transfers																										
da0: 7502MB (15364416 512 byte sectors)																							
da0: quirks=0x2<NO_6_BYTE>																										
da0: Delete methods: <NONE(*),ZERO>																								
pass2: <Kingston DataTraveler 3.0 PMAP> Removable Direct Access SPC-4 SCSI device												
pass2: Serial Number 0019E06B9C90BE61376B7786																					
pass2: 40.000MB/s transfers																										
ada0: <Hitachi HUA721010KLA330 GKAOA70M> ATA-7 SATA 1.x device																	
ada0: Serial Number GTA160PBKBYVMF																								
ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)																		
ada0: 953869MB (1953525168 512 byte sectors)																						
ada0: Previously was known as ad4																								
ada1: <Hitachi HUA721010KLA330 GKAOAB0A> ATA-7 SATA 2.x device																	
ada1: Serial Number GTE002PBJ52EUE																								
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)																		
ada1: 953869MB (1953525168 512 byte sectors)																						
ada1: Previously was known as ad6


Now I have something to work with... my plan of action,

a) Use Hitachi's feature change util to change to SATA2 (1 test drive)

b) Cross flash my backup LSI2008 with another firmware (TBD) that will allow me to change port settings/address this silly thing.

Again, I can't thank you enough for your help with this... I will post updates with my progress, hopefully later tonight.

Mike
 
Joined
Oct 24, 2016
Messages
10
I used HGST's "Feature Tool" (avail. here) to change 1 drive's SATA operating mode to 2 - 300MB/s

Currently booting FreeNAS, and for the first time since the card firmware upgrade the drives are spinning up when plugged into LSI2008-IT (P20)

Not only is it spinning, FreeNAS sees it... holy f (pardon my french)!

Updates to follow after I do the rest of the drives and re-boot into FreeNAS

Mike
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,862
Joined
Oct 24, 2016
Messages
10
Well ladies & gents..... winner, winner chicken dinner!

One thing I have heard happens with certain SAS controllers, is that they won't recognize SATA I disk speed, (1.5Gbps).
SAS I is 3Gbps, SAS II is 6Gbps and SAS III is 12Gbps. ]
All 5 problematic Hitachi drives changed to SATAII - 3GBps, and as shown below they now appear in FreeNAS.
Code:
# camcontrol devlist											
<ATA Hitachi HUA72101 AB0A>		at scbus0 target 1 lun 0 (pass0,da0)		
<ATA Hitachi HUA72101 A70M>		at scbus0 target 2 lun 0 (pass1,da1)		
<ATA Hitachi HUA72101 A70M>		at scbus0 target 3 lun 0 (pass2,da2)		
<ATA Hitachi HUA72101 A70M>		at scbus0 target 4 lun 0 (pass3,da3)		
<ATA Hitachi HUA72101 A70M>		at scbus0 target 5 lun 0 (pass4,da4)		
<Hitachi HUA721010KLA330 GKAOAB0A>  at scbus1 target 0 lun 0 (ada0,pass5)	 
<Hitachi HUA721010KLA330 GKAOA70M>  at scbus2 target 0 lun 0 (ada1,pass6)	 
<Kingston DataTraveler 3.0 PMAP>   at scbus4 target 0 lun 0 (pass7,da5)		
<ATEN Virtual CDROM YS0J>		  at scbus5 target 0 lun 0 (pass8,cd0)

The 1x Seagate is still down, and I can confirm the Hitachi tool (linked above) is not compatible to make this change (LOL - I wonder why :p) : given how many spare Hitachi drives I have available, I figured theres no harm in potentially bricking this Seagate. Damn HGST put validation in place to prevent this sort of nonsense...

I then attempted SeaTools (Avail. here) but it doesn't allow changing the operating mode, simply testing the device. Per a few other/rando forum posts I tried different combinations of jumper settings, as the default (no jumper) is apparently suppose to be SATA2-300MBps, clearly wasn't working.

I intend to install one of my many spare hitachis and call it a day, thank you everyone for your help!

Mike

EDIT: The 2 drives currently plugged in via Intel are the 2 drives that ALWAYS worked, I simply was too impatient to test and deferred rewiring them into the LSI card
 
Last edited by a moderator:

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,862

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,108
Don't feel bad. I figured it out by accident after months. :)
And it doesn't work with the fancy new editor I'm testing. I've been needing to save files, then drag and drop them and finally attach them as full images whenever I want to place an image.
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,862
And it doesn't work with the fancy new editor I'm testing. I've been needing to save files, then drag and drop them and finally attach them as full images whenever I want to place an image.
That's not good
 

Arwen

FreeNAS Expert
Joined
May 17, 2014
Messages
1,142
Well ladies & gents..... winner, winner chicken dinner!

All 5 problematic Hitachi drives changed to SATAII - 3GBps, and as shown below they now appear in FreeNAS.
...
I intend to install one of my many spare hitachis and call it a day, thank you everyone for your help!

Mike
Glad I could help. You are welcome.

Last, there is no problem per say to having some disks on the LSI and some on system board SATA ports. Just a
mater taste, and occasionally reliability. For example Marvell SATA system board ports would be less reliable than
your LSI, (or Intel SATA).
 
Top