Disk detection order

[NachoMan77]

Hi,

maybe this is a newbie question, or maybe not... so here it goes...

I have 4 x 3TB WD Reds connected to a M1015 (flashed in IR mode). Drives are unconfigured on the controller (ie: no RAID).

Also, I'm runnig a virtual FreeNAS, with passthrough to the M1015.

I'd like to know why are the drives not being detected in the right port order? I already tried with a different SATA breakout cable (with port labels) and I still get the same result.

port#__ detected as:
1 --------- da4
2 --------- da2
3 --------- da1
4 --------- da3

Any way to force an ordered detection?

Thanks.

 

[jgreco]

Interesting question. Is this being probed as an "mps" controller, or "mpt"? I don't know if there's a userland utility for debugging "mps" attached disks. For mpt, try "mfiutil".

 

[NachoMan77]

mps0: <LSI SAS2008> port 0x4000-0x40ff mem 0xfd4fc000-0xfd4fffff,0xfd480000-0xfd4bffff irq 18 at device 0.0 on pci3
mps0: Firmware: 16.00.00.00, Driver: 14.00.00.01-fbsd
mps0: IOCCapabilities: 185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR>

Tried reseting the card to its default values and disabled staggered spin-up, but nothing changed.

 

[NachoMan77]

Code:

da3 at mps0 bus 0 scbus3 target 10 lun 0
da3: <ATA WDC WD30EFRX-68A 0A80> Fixed Direct Access SCSI-6 device
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da3: quirks=0x8<4K>
da4 at mps0 bus 0 scbus3 target 11 lun 0
da4: <ATA WDC WD30EFRX-68A 0A80> Fixed Direct Access SCSI-6 device
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da4: quirks=0x8<4K>
Timecounter "TSC" frequency 1795841000 Hz quality 1000
da1 at mps0 bus 0 scbus3 target 8 lun 0
da1: <ATA WDC WD30EFRX-68A 0A80> Fixed Direct Access SCSI-6 device
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da1: quirks=0x8<4K>
da2 at mps0 bus 0 scbus3 target 9 lun 0
da2: <ATA WDC WD30EFRX-68A 0A80> Fixed Direct Access SCSI-6 device
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da2: quirks=0x8<4K>

Code:

[root@freenas] ~# camcontrol devlist
<NECVMWar VMware IDE CDR10 1.00>  at scbus1 target 0 lun 0 (pass0,cd0)
<VMware Virtual disk 1.0>          at scbus2 target 0 lun 0 (da0,pass1)
<ATA WDC WD30EFRX-68A 0A80>        at scbus3 target 8 lun 0 (da1,pass2)
<ATA WDC WD30EFRX-68A 0A80>        at scbus3 target 9 lun 0 (da2,pass3)
<ATA WDC WD30EFRX-68A 0A80>        at scbus3 target 10 lun 0 (da3,pass4)
<ATA WDC WD30EFRX-68A 0A80>        at scbus3 target 11 lun 0 (da4,pass5)

The strange thing is that I'm using physical Port#1 of M1015, which according to LSI BIOS maps to slots (targets?) 0,1,2 and 3. Likewise, Port#0 maps to slots 4,5,6 and 7. So there's the first quirk, physical ports appear to be swapped.

Next thing, according to dmesg, da3 is probed first (physical disk #4), then da4 (disk #1), then da1 (disk #3) and finally da2 (disk #2).

Strange indeed.

 

[jgreco]

So what does it show in the BIOS (thinking: SAS topology). Remember, what you have is actually a SAS controller, not SATA. It is designed to handle some abstraction. I don't think FreeNAS includes smp_utils yet, which might also be helpful.

 

[jgreco]

oh and p.s. I suspect rearranging the drives would work.

 

[NachoMan77]

Ok, I made some progress...

I sourced an LSI utility aptly called lsiutil which provided some insight.

Apparently these controllers have some sort of memory for disk mappings. I used lsiutil to erase these mappings using the following instructions:

http://lists.freebsd.org/pipermail/freebsd-scsi/2012-February/005252.html

LSIUtil 1.63: http://www.juhonkoti.net/media/LSIUTIL-1.63.zip
LSIUtil Manual: http://www.thomas-krenn.com/de/wikiDE/images/4/44/Lsi_userguide_2006_20130528.pdf

I also changed this option:
Option 13: Change SAS IO Unit setting

and manually assigned the PortIds to match their PHYNum.

Now at least I get them detected in order... but not in the right order. Instead of getting 1,2,3,4; the disks get probed as 4,3,2,1

Code:

[root@freenas] ~# camcontrol devlist
<NECVMWar VMware IDE CDR10 1.00>  at scbus1 target 0 lun 0 (pass0,cd0)
<VMware Virtual disk 1.0>          at scbus2 target 0 lun 0 (da0,pass1)
<ATA WDC WD30EFRX-68A 0A80>        at scbus3 target 6 lun 0 (da1,pass2)
<ATA WDC WD30EFRX-68A 0A80>        at scbus3 target 7 lun 0 (da2,pass3)
<ATA WDC WD30EFRX-68A 0A80>        at scbus3 target 8 lun 0 (da3,pass4)
<ATA WDC WD30EFRX-68A 0A80>        at scbus3 target 9 lun 0 (da4,pass5)

Also, the target # shifted a little bit.

Apparently this is more of an LSI firmware/driver thing than a FreeNAS issue. I'll drop it for now.