SAS 9300-8i dual linked to BPN-SAS3-826EL1 = I/O errors on all SATA SSDs ?

Lars Jensen · Oct 9, 2020

Hi all

Got a Broadcom SAS 9300-8i (with SAS3008) HBA with the latest firmware where I get I/O errors on Micron 5300 MAX SSDs when connecting both ports on the HBA to the BPN-SAS3-826EL1 backplane in a Supermicro server.

If I unplug one of the cables on the HBA no errors occur.

I haven't experienced this before with SAS2308. Does SAS3008 do some kind of multipathing that doesn't work with SATA connected drives on a SAS (and SATA compatible) backplane?

I'm hoping there's a configuration setting somewhere which could solve this, or I will need to remember to leave one of the cables detached

sretalla · Oct 9, 2020

When you say latest firmware, do you mean the one from @JoshDW19 ?

There's a specific patch applicable to SSDs and 9300s.

LSI 9300-xx Firmware Update

JoshDW19 submitted a new resource: LSI 9300-xx Firmware Update - Fixes a controller reset issue in LSI 9300-xx HBA cards Hey Community, If you are using an LSI 9300 HBA with FreeNAS or the soon-to-be TrueNAS CORE, you may experience some performance issues causing the controller to reset...

www.ixsystems.com

Lars Jensen · Oct 9, 2020

Firmware is the current latest from Broadcom site 16.00.10.00. Didn't know the other one existed

Do you know if 16.00.12.00 fixes performance issues only, or also I/O errors with connected SATA drives?

I can try 16.00.12.00 to see if it fixes the issue, but have to make a little trip to physically plug in the cable again.

sretalla · Oct 9, 2020

Lars Jensen said:
Do you know if 16.00.12.00 fixes performance issues only, or also I/O errors with connected SATA drives?

I only know that .12 was specifically for an SSD reset issue with those controllers... not sure what was in the .11 version.

Johnnie Black · Oct 9, 2020

Lars Jensen said:
Does SAS3008 do some kind of multipathing that doesn't work with SATA connected drives on a SAS (and SATA compatible) backplane?

No, I have one working with the same expander and SATA SSDs in dual link without issues (with Linux though, but the OS is not likely the problem), possibly some compatibilty issues with the SSDs.

Lars Jensen · Oct 13, 2020

Just a quick update.

Using 16.00.12.00 didn't solve the problem.

Also tried some older Samsung SM853T SSD which gave the same error.

SATA is only on one channel, so maybe the SAS 9300-8i just needs only one link and not get confused.

Johnnie Black · Oct 13, 2020

Lars Jensen said:
so maybe the SAS 9300-8i just needs only one link and not get confused.

Like mentioned I have no issues using dual link with same HBA and expander.

Lars Jensen · Oct 14, 2020

Johnnie Black said:
Like mentioned I have no issues using dual link with same HBA and expander.

Which firmware version do you use?

I've tested 16.00.10.00 and 16.00.12.00.

I'll try testing with SAS disks later to see if that works.

Johnnie Black · Oct 14, 2020

I'll ckeck later when home, and since that hardware is currently in a test server I'll see if I have some time to install TrueNAS and test there, do you just need to boot to see the errors or only after some file transfer or scrub?

Lars Jensen · Oct 14, 2020

Yes, when booting errors keep coming.

Johnnie Black · Oct 15, 2020

Did a quick test booting TrueNAS 12.0-RC1, all seems to be working correctly, there are 12 small SATA SSDs connected, and you can see it's using dual link:

Code:

camcontrol smpphylist /dev/ses0
31 PHYs:
PHY  Attached SAS Address
  0  0x500304801ef3db80   <ATA TS32GSSD370S 4H>             (pass8,da8)
  1  0x500304801ef3db81   <ATA TS32GSSD370S 4H>             (pass9,da9)
  2  0x500304801ef3db82   <ATA TS32GSSD370S 4H>             (pass10,da10)
  3  0x500304801ef3db83   <ATA TS32GSSD370S 4H>             (pass11,da11)
  4  0x500304801ef3db84   <ATA TS32GSSD370S 4H>             (pass12,da12)
  5  0x500304801ef3db85   <ATA TS32GSSD370S 4H>             (pass13,da13)
  6  0x500304801ef3db86   <ATA TS32GSSD370S 4H>             (pass14,da14)
  7  0x500304801ef3db87   <ATA TS32GSSD370S 4H>             (pass15,da15)
  8  0x500304801ef3db88   <ATA TS32GSSD370S 4H>             (pass16,da16)
  9  0x500304801ef3db89   <ATA TS32GSSD370S 2B>             (pass17,da17)
10  0x500304801ef3db8a   <ATA TS32GSSD370S 4H>             (pass18,da18)
11  0x500304801ef3db8b   <ATA TS32GSSD370S 4H>             (pass19,da19)
12  0x0000000000000000
13  0x0000000000000000
14  0x0000000000000000
15  0x0000000000000000
16  0x0000000000000000
17  0x0000000000000000
18  0x0000000000000000
19  0x0000000000000000
20  0x500605b00b1c2d00
21  0x500605b00b1c2d00
22  0x500605b00b1c2d00
23  0x500605b00b1c2d00
24  0x500605b00b1c2d00
25  0x500605b00b1c2d00
26  0x500605b00b1c2d00
27  0x500605b00b1c2d00
28  0x500304801ef3dbbd   <LSI SAS3x28 0601>                (pass20,ses0)
29  0x0000000000000000
30  0x0000000000000000

HBAs (didn't check which one is connected to the expander but they are using the same firmware):

Code:

 sas3flash -listall
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

        Adapter Selected is a Avago SAS: SAS3008(C0)

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS3008(C0)  16.00.01.00    0e.01.00.07    08.37.00.00     00:01:00:00
1  SAS3008(C0)  16.00.01.00    0e.01.00.07    08.37.00.00     00:02:00:00

        Finished Processing Commands Successfully.
        Exiting SAS3Flash.

Full dmesg here:

Code:

FreeBSD 12.2-PRERELEASE 4912790fb32(HEAD) TRUENAS

        TrueNAS (c) 2009-2020, iXsystems, Inc.
        All rights reserved.
        TrueNAS code is released under the modified BSD license with some
        files copyrighted by (c) iXsystems, Inc.

        For more information, documentation, help or support, go here:
        http://truenas.com
Welcome to TrueNAS

Warning: settings changed through the CLI are not written to
the configuration database and will be reset on reboot.

root@truenas[~]# dmesg
---<<BOOT>>---
Copyright (c) 1992-2020 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.2-PRERELEASE 4912790fb32(HEAD) TRUENAS amd64
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
VT(vga): text 80x25
CPU: Intel(R) Pentium(R) CPU G4560 @ 3.50GHz (3504.12-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906e9  Family=0x6  Model=0x9e  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4ffaebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x2946687<FSGSBASE,TSCADJ,SGX,SMEP,ERMS,INVPCID,NFPUSG,MPX,RDSEED,SMAP,CLFLUSHOPT,PROCTRACE>
  Structured Extended Features3=0x9c000000<IBPB,STIBP,L1DFL,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8053985280 (7680 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: < >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
arc4random: no preloaded entropy cache
ioapic0 <Version 2.0> irqs 0-23 on motherboard
Launching APs: 1 3 2
Timecounter "TSC-low" frequency 1752058717 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
mlx5en: Mellanox Ethernet driver 3.5.2 (September 2019)
nexus0
vtvga0: <VT VGA driver> on motherboard
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard
padlock0: No ACE support.
cryptosoft0: <software crypto> on motherboard
acpi0: <SUPERM SUPERM> on motherboard
acpi0: Power Button (fixed)
unknown: memory range not supported
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 550
Event timer "HPET1" frequency 24000000 Hz quality 440
Event timer "HPET2" frequency 24000000 Hz quality 440
Event timer "HPET3" frequency 24000000 Hz quality 440
Event timer "HPET4" frequency 24000000 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
apei0: <ACPI Platform Error Interface> on acpi0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
mpr0: <Avago Technologies (LSI) SAS3008> port 0xe000-0xe0ff mem 0xdf440000-0xdf44ffff,0xdf400000-0xdf43ffff irq 16 at device 0.0 on pci1
mpr0: Firmware: 16.00.01.00, Driver: 23.00.00.00-fbsd
mpr0: IOCCapabilities: 7a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc,FastPath,RDPQArray>
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 1.1 on pci0
pci2: <ACPI PCI bus> on pcib2
mpr1: <Avago Technologies (LSI) SAS3008> port 0xd000-0xd0ff mem 0xdf240000-0xdf24ffff,0xdf200000-0xdf23ffff irq 17 at device 0.0 on pci2
mpr1: Firmware: 16.00.01.00, Driver: 23.00.00.00-fbsd
mpr1: IOCCapabilities: 7a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc,FastPath,RDPQArray>
pci0: <old, non-VGA display device> at device 19.0 (no driver attached)
xhci0: <Intel Sunrise Point USB 3.0 controller> mem 0xdf800000-0xdf80ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <Intel Sunrise Point AHCI SATA controller> port 0xf050-0xf057,0xf040-0xf043,0xf020-0xf03f mem 0xdf810000-0xdf811fff,0xdf81e000-0xdf81e0ff,0xdf81d000-0xdf81d7ff irq 16 at device 23.0 on pci0
ahci0: AHCI v1.31 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci3: <ACPI PCI bus> on pcib3
ahci1: <JMicron JMB58x AHCI SATA controller> port 0xc200-0xc27f,0xc180-0xc1ff,0xc100-0xc17f,0xc080-0xc0ff,0xc000-0xc07f mem 0xdf710000-0xdf711fff irq 16 at device 0.0 on pci3
ahci1: AHCI v1.31 with 5 6Gbps ports, Port Multiplier supported with FBS
ahcich6: <AHCI channel> at channel 0 on ahci1
ahcich7: <AHCI channel> at channel 1 on ahci1
(noperiph:ahcich6:0:-1:ffffffff): rescan already queued
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 29.0 on pci0
pci4: <ACPI PCI bus> on pcib4
igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xb000-0xb01f mem 0xdf600000-0xdf67ffff,0xdf680000-0xdf683fff irq 16 at device 0.0 on pci4
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 2 RX queues 2 TX queues
igb0: Using MSI-X interrupts with 3 vectors
igb0: Ethernet address: ac:1f:6b:f0:33:36
pcib5: <ACPI PCI-PCI bridge> irq 17 at device 29.1 on pci0
pci5: <ACPI PCI bus> on pcib5
igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xa000-0xa01f mem 0xdf500000-0xdf57ffff,0xdf580000-0xdf583fff irq 17 at device 0.0 on pci5
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 2 RX queues 2 TX queues
igb1: Using MSI-X interrupts with 3 vectors
igb1: Ethernet address: ac:1f:6b:f0:33:37
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 29.2 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 0.0 on pci6
pci7: <ACPI PCI bus> on pcib7
vgapci0: <VGA-compatible display> port 0x9000-0x907f mem 0xde000000-0xdeffffff,0xdf000000-0xdf01ffff irq 18 at device 0.0 on pci7
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <memory> at device 31.2 (no driver attached)
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
ipmi0: <IPMI System Interface> port 0xca2,0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xd1fff,0xd2000-0xd2fff pnpid ORM0000 on isa0
arc4random: no preloaded entropy cache
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
arc4random: no preloaded entropy cache
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ugen0.1: <0x8086 XHCI root HUB> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ipmi0: IPMI device rev. 1, firmware rev. 1.58, version 2.0, device support mask 0xbf
ipmi0: Number of channels 2
ipmi0: Attached watchdog
ipmi0: Establishing power cycle handler
random: unblocking device.
arc4random: no preloaded entropy cache
Trying to mount root from zfs:boot-pool/ROOT/default []...
Root mount waiting for: CAM usbus0
Root mount waiting for: CAM usbus0
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x0009> enclosureHandle<0x0001> slot 3
mpr0: At enclosure level 0 and connector name (    )
uhub0: 22 ports with 22 removable, self powered
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x000a> enclosureHandle<0x0001> slot 2
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x000b> enclosureHandle<0x0001> slot 0
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x000c> enclosureHandle<0x0001> slot 1
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x000d> enclosureHandle<0x0001> slot 7
mpr0: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 2
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000d> enclosureHandle<0x0002> slot 3
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000e> enclosureHandle<0x0002> slot 4
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000f> enclosureHandle<0x0002> slot 5
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0010> enclosureHandle<0x0002> slot 6
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0011> enclosureHandle<0x0002> slot 7
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0012> enclosureHandle<0x0002> slot 8
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0013> enclosureHandle<0x0002> slot 9
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0014> enclosureHandle<0x0002> slot 10
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0015> enclosureHandle<0x0002> slot 11
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x0016> enclosureHandle<0x0002> slot 12
mpr1: At enclosure level 0 and connector name (    )
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x000e> enclosureHandle<0x0001> slot 6
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x000f> enclosureHandle<0x0001> slot 4
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x0010> enclosureHandle<0x0001> slot 5
mpr0: At enclosure level 0 and connector name (    )
Root mount waiting for: CAM usbus0
ugen0.2: <vendor 0x0557 product 0x7000> at usbus0
uhub1 on uhub0
uhub1: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 1> on usbus0
uhub1: 4 ports with 3 removable, self powered
Root mount waiting for: CAM usbus0
ugen0.3: <vendor 0x0557 product 0x2419> at usbus0
ukbd0 on uhub1
ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 2> on usbus0
kbd2 at ukbd0
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
ses0 at mpr1 bus 0 scbus1 target 20 lun 0
ses0: <LSI SAS3x28 0601> Fixed Enclosure Services SPC-3 SCSI device
ses0: 1200.000MB/s transfers
ses0: Command Queueing enabled
ses0: SES Device
ses1 at ahciem0 bus 0 scbus8 target 0 lun 0
ses1: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses1: SEMB SES Device
ada0 at ahcich1 bus 0 scbus3 target 0 lun 0
ada0: <TS128GSSD370S P1225CH1> ACS-2 ATA SATA 3.x device
ada0: Serial Number E862862409
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes)
ada0: Command Queueing enabled
ada0: 122104MB (250069680 512 byte sectors)
ada1 at ahcich2 bus 0 scbus4 target 0 lun 0
ada1: <TS120GSSD220S S0628A0> ACS-2 ATA SATA 3.x device
ada1: Serial Number F853320329
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada1: Command Queueing enabled
ada1: 114473MB (234441648 512 byte sectors)
ses1: ada0 in 'Slot 01', SATA Slot: scbus3 target 0
ses1: ada1 in 'Slot 02', SATA Slot: scbus4 target 0
ada2 at ahcich3 bus 0 scbus5 target 0 lun 0
ada2: <TS120GSSD220S S0628A0> ACS-2 ATA SATA 3.x device
ada2: Serial Number F853320393
ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada2: Command Queueing enabled
ada2: 114473MB (234441648 512 byte sectors)
ses1: ada2 in 'Slot 03', SATA Slot: scbus5 target 0
ada3 at ahcich6 bus 0 scbus9 target 0 lun 0
ada3: <TS120GSSD220S S0628A0> ACS-2 ATA SATA 3.x device
ada3: Serial Number F853320399
ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada3: Command Queueing enabled
ada3: 114473MB (234441648 512 byte sectors)
ada4 at ahcich7 bus 0 scbus10 target 0 lun 0
ada4: <TS120GSSD220S S0628A0> ACS-2 ATA SATA 3.x device
ada4: Serial Number F853320398
ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada4: Command Queueing enabled
ada4: 114473MB (234441648 512 byte sectors)
ses0: da8,pass8 in 'Slot00', SAS Slot: 1 phys at slot 0
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db80
ses0: da9,pass9 in 'Slot01', SAS Slot: 1 phys at slot 1
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db81
ses0: da10,pass10 in 'Slot02', SAS Slot: 1 phys at slot 2
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db82
ses0: da11,pass11 in 'Slot03', SAS Slot: 1 phys at slot 3
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db83
ses0: da12,pass12 in 'Slot04', SAS Slot: 1 phys at slot 4
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db84
ses0: da13,pass13 in 'Slot05', SAS Slot: 1 phys at slot 5
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db85
ses0: da14,pass14 in 'Slot06', SAS Slot: 1 phys at slot 6
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db86
ses0: da15,pass15 in 'Slot07', SAS Slot: 1 phys at slot 7
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db87
ses0: da16,pass16 in 'Slot08', SAS Slot: 1 phys at slot 8
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db88
ses0: da17,pass17 in 'Slot09', SAS Slot: 1 phys at slot 9
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db89
ses0: da18,pass18 in 'Slot10', SAS Slot: 1 phys at slot 10
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db8a
ses0: da19,pass19 in 'Slot11', SAS Slot: 1 phys at slot 11
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db8b
da17 at mpr1 bus 0 scbus1 target 17 lun 0
da17: <ATA TS32GSSD370S 2B> Fixed Direct Access SPC-4 SCSI device
da17: Serial Number F758460614
da17: 1200.000MB/s transfers
da17: Command Queueing enabled
da17: 30533MB (62533296 512 byte sectors)
da8 at mpr1 bus 0 scbus1 target 8 lun 0
da8: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da8: Serial Number C160930075
da8: 1200.000MB/s transfers
da8: Command Queueing enabled
da8: 30533MB (62533296 512 byte sectors)
da1 at mpr0 bus 0 scbus0 target 1 lun 0
da1: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number C571900873
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 61057MB (125045424 512 byte sectors)
da0 at mpr0 bus 0 scbus0 target 0 lun 0
da0: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number C571900869
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 61057MB (125045424 512 byte sectors)
da3 at mpr0 bus 0 scbus0 target 3 lun 0
da3: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number C571900885
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 61057MB (125045424 512 byte sectors)
da2 at mpr0 bus 0 scbus0 target 2 lun 0
da2: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number C571900876
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 61057MB (125045424 512 byte sectors)
da10 at mpr1 bus 0 scbus1 target 10 lun 0
da10: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da10: Serial Number C364420142
da10: 1200.000MB/s transfers
da10: Command Queueing enabled
da10: 30533MB (62533296 512 byte sectors)
da6 at mpr0 bus 0 scbus0 target 6 lun 0
da6: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number C571900866
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 61057MB (125045424 512 byte sectors)
da4 at mpr0 bus 0 scbus0 target 4 lun 0
da4: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number C571900855
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 61057MB (125045424 512 byte sectors)
da5 at mpr0 bus 0 scbus0 target 5 lun 0
da5: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number C571900865
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 61057MB (125045424 512 byte sectors)
da11 at mpr1 bus 0 scbus1 target 11 lun 0
da11: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da11: Serial Number C364420148
da11: 1200.000MB/s transfers
da11: Command Queueing enabled
da11: 30533MB (62533296 512 byte sectors)
da12 at mpr1 bus 0 scbus1 target 12 lun 0
da12: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da12: Serial Number C333750271
da12: 1200.000MB/s transfers
da12: Command Queueing enabled
da12: 30533MB (62533296 512 byte sectors)
da7 at mpr0 bus 0 scbus0 target 7 lun 0
da7: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number C571900867
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 61057MB (125045424 512 byte sectors)
da13 at mpr1 bus 0 scbus1 target 13 lun 0
da13: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da13: Serial Number C392050392
da13: 1200.000MB/s transfers
da13: Command Queueing enabled
da13: 30533MB (62533296 512 byte sectors)
da18 at mpr1 bus 0 scbus1 target 18 lun 0
da18: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da18: Serial Number C160931700
da18: 1200.000MB/s transfers
da18: Command Queueing enabled
da18: 30533MB (62533296 512 byte sectors)
da14 at mpr1 bus 0 scbus1 target 14 lun 0
da14: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da14: Serial Number C392050400
da14: 1200.000MB/s transfers
da14: Command Queueing enabled
da14: 30533MB (62533296 512 byte sectors)
da9 at mpr1 bus 0 scbus1 target 9 lun 0
da9: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da9: Serial Number C364420131
da9: 1200.000MB/s transfers
da9: Command Queueing enabled
da9: 30533MB (62533296 512 byte sectors)
da16 at mpr1 bus 0 scbus1 target 16 lun 0
da16: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da16: Serial Number C364420439
da16: 1200.000MB/s transfers
da16: Command Queueing enabled
da16: 30533MB (62533296 512 byte sectors)
da15 at mpr1 bus 0 scbus1 target 15 lun 0
da15: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da15: Serial Number C364420430
da15: 1200.000MB/s transfers
da15: Command Queueing enabled
da15: 30533MB (62533296 512 byte sectors)
da19 at mpr1 bus 0 scbus1 target 19 lun 0
da19: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da19: Serial Number C304332037
da19: 1200.000MB/s transfers
da19: Command Queueing enabled
da19: 30533MB (62533296 512 byte sectors)
ums0 on uhub1
ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 2> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
lo0: link state changed to UP
igb0: link state changed to UP
GEOM_ELI: Device ada0p3.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: hardware
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI>
igb0: link state changed to DOWN
igb0: link state changed to UP
CPU: Intel(R) Pentium(R) CPU G4560 @ 3.50GHz (3504.12-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906e9  Family=0x6  Model=0x9e  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4ffaebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x2946687<FSGSBASE,TSCADJ,SGX,SMEP,ERMS,INVPCID,NFPUSG,MPX,RDSEED,SMAP,CLFLUSHOPT,PROCTRACE>
  Structured Extended Features3=0x9c002400<MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
Security policy loaded: MAC/ntpd (mac_ntpd)
pid 2174 (syslog-ng), jid 0, uid 0: exited on signal 6 (core dumped)

Lars Jensen · Oct 15, 2020

Johnnie Black said:

Did a quick test booting TrueNAS 12.0-RC1, all seems to be working correctly, there are 12 small SATA SSDs connected, and you can see it's using dual link:

Code:

camcontrol smpphylist /dev/ses0
31 PHYs:
PHY  Attached SAS Address
  0  0x500304801ef3db80   <ATA TS32GSSD370S 4H>             (pass8,da8)
  1  0x500304801ef3db81   <ATA TS32GSSD370S 4H>             (pass9,da9)
  2  0x500304801ef3db82   <ATA TS32GSSD370S 4H>             (pass10,da10)
  3  0x500304801ef3db83   <ATA TS32GSSD370S 4H>             (pass11,da11)
  4  0x500304801ef3db84   <ATA TS32GSSD370S 4H>             (pass12,da12)
  5  0x500304801ef3db85   <ATA TS32GSSD370S 4H>             (pass13,da13)
  6  0x500304801ef3db86   <ATA TS32GSSD370S 4H>             (pass14,da14)
  7  0x500304801ef3db87   <ATA TS32GSSD370S 4H>             (pass15,da15)
  8  0x500304801ef3db88   <ATA TS32GSSD370S 4H>             (pass16,da16)
  9  0x500304801ef3db89   <ATA TS32GSSD370S 2B>             (pass17,da17)
10  0x500304801ef3db8a   <ATA TS32GSSD370S 4H>             (pass18,da18)
11  0x500304801ef3db8b   <ATA TS32GSSD370S 4H>             (pass19,da19)
12  0x0000000000000000
13  0x0000000000000000
14  0x0000000000000000
15  0x0000000000000000
16  0x0000000000000000
17  0x0000000000000000
18  0x0000000000000000
19  0x0000000000000000
20  0x500605b00b1c2d00
21  0x500605b00b1c2d00
22  0x500605b00b1c2d00
23  0x500605b00b1c2d00
24  0x500605b00b1c2d00
25  0x500605b00b1c2d00
26  0x500605b00b1c2d00
27  0x500605b00b1c2d00
28  0x500304801ef3dbbd   <LSI SAS3x28 0601>                (pass20,ses0)
29  0x0000000000000000
30  0x0000000000000000

HBAs (didn't check which one is connected to the expander but they are using the same firmware):

Code:

 sas3flash -listall
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

        Adapter Selected is a Avago SAS: SAS3008(C0)

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS3008(C0)  16.00.01.00    0e.01.00.07    08.37.00.00     00:01:00:00
1  SAS3008(C0)  16.00.01.00    0e.01.00.07    08.37.00.00     00:02:00:00

        Finished Processing Commands Successfully.
        Exiting SAS3Flash.

Full dmesg here:

Code:

FreeBSD 12.2-PRERELEASE 4912790fb32(HEAD) TRUENAS

        TrueNAS (c) 2009-2020, iXsystems, Inc.
        All rights reserved.
        TrueNAS code is released under the modified BSD license with some
        files copyrighted by (c) iXsystems, Inc.

        For more information, documentation, help or support, go here:
        http://truenas.com
Welcome to TrueNAS

Warning: settings changed through the CLI are not written to
the configuration database and will be reset on reboot.

root@truenas[~]# dmesg
---<<BOOT>>---
Copyright (c) 1992-2020 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.2-PRERELEASE 4912790fb32(HEAD) TRUENAS amd64
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
VT(vga): text 80x25
CPU: Intel(R) Pentium(R) CPU G4560 @ 3.50GHz (3504.12-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906e9  Family=0x6  Model=0x9e  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4ffaebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x2946687<FSGSBASE,TSCADJ,SGX,SMEP,ERMS,INVPCID,NFPUSG,MPX,RDSEED,SMAP,CLFLUSHOPT,PROCTRACE>
  Structured Extended Features3=0x9c000000<IBPB,STIBP,L1DFL,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8053985280 (7680 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: < >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
arc4random: no preloaded entropy cache
ioapic0 <Version 2.0> irqs 0-23 on motherboard
Launching APs: 1 3 2
Timecounter "TSC-low" frequency 1752058717 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
mlx5en: Mellanox Ethernet driver 3.5.2 (September 2019)
nexus0
vtvga0: <VT VGA driver> on motherboard
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard
padlock0: No ACE support.
cryptosoft0: <software crypto> on motherboard
acpi0: <SUPERM SUPERM> on motherboard
acpi0: Power Button (fixed)
unknown: memory range not supported
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 550
Event timer "HPET1" frequency 24000000 Hz quality 440
Event timer "HPET2" frequency 24000000 Hz quality 440
Event timer "HPET3" frequency 24000000 Hz quality 440
Event timer "HPET4" frequency 24000000 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
apei0: <ACPI Platform Error Interface> on acpi0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
mpr0: <Avago Technologies (LSI) SAS3008> port 0xe000-0xe0ff mem 0xdf440000-0xdf44ffff,0xdf400000-0xdf43ffff irq 16 at device 0.0 on pci1
mpr0: Firmware: 16.00.01.00, Driver: 23.00.00.00-fbsd
mpr0: IOCCapabilities: 7a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc,FastPath,RDPQArray>
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 1.1 on pci0
pci2: <ACPI PCI bus> on pcib2
mpr1: <Avago Technologies (LSI) SAS3008> port 0xd000-0xd0ff mem 0xdf240000-0xdf24ffff,0xdf200000-0xdf23ffff irq 17 at device 0.0 on pci2
mpr1: Firmware: 16.00.01.00, Driver: 23.00.00.00-fbsd
mpr1: IOCCapabilities: 7a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc,FastPath,RDPQArray>
pci0: <old, non-VGA display device> at device 19.0 (no driver attached)
xhci0: <Intel Sunrise Point USB 3.0 controller> mem 0xdf800000-0xdf80ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <Intel Sunrise Point AHCI SATA controller> port 0xf050-0xf057,0xf040-0xf043,0xf020-0xf03f mem 0xdf810000-0xdf811fff,0xdf81e000-0xdf81e0ff,0xdf81d000-0xdf81d7ff irq 16 at device 23.0 on pci0
ahci0: AHCI v1.31 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci3: <ACPI PCI bus> on pcib3
ahci1: <JMicron JMB58x AHCI SATA controller> port 0xc200-0xc27f,0xc180-0xc1ff,0xc100-0xc17f,0xc080-0xc0ff,0xc000-0xc07f mem 0xdf710000-0xdf711fff irq 16 at device 0.0 on pci3
ahci1: AHCI v1.31 with 5 6Gbps ports, Port Multiplier supported with FBS
ahcich6: <AHCI channel> at channel 0 on ahci1
ahcich7: <AHCI channel> at channel 1 on ahci1
(noperiph:ahcich6:0:-1:ffffffff): rescan already queued
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 29.0 on pci0
pci4: <ACPI PCI bus> on pcib4
igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xb000-0xb01f mem 0xdf600000-0xdf67ffff,0xdf680000-0xdf683fff irq 16 at device 0.0 on pci4
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 2 RX queues 2 TX queues
igb0: Using MSI-X interrupts with 3 vectors
igb0: Ethernet address: ac:1f:6b:f0:33:36
pcib5: <ACPI PCI-PCI bridge> irq 17 at device 29.1 on pci0
pci5: <ACPI PCI bus> on pcib5
igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xa000-0xa01f mem 0xdf500000-0xdf57ffff,0xdf580000-0xdf583fff irq 17 at device 0.0 on pci5
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 2 RX queues 2 TX queues
igb1: Using MSI-X interrupts with 3 vectors
igb1: Ethernet address: ac:1f:6b:f0:33:37
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 29.2 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 0.0 on pci6
pci7: <ACPI PCI bus> on pcib7
vgapci0: <VGA-compatible display> port 0x9000-0x907f mem 0xde000000-0xdeffffff,0xdf000000-0xdf01ffff irq 18 at device 0.0 on pci7
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <memory> at device 31.2 (no driver attached)
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
ipmi0: <IPMI System Interface> port 0xca2,0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xd1fff,0xd2000-0xd2fff pnpid ORM0000 on isa0
arc4random: no preloaded entropy cache
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
arc4random: no preloaded entropy cache
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ugen0.1: <0x8086 XHCI root HUB> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ipmi0: IPMI device rev. 1, firmware rev. 1.58, version 2.0, device support mask 0xbf
ipmi0: Number of channels 2
ipmi0: Attached watchdog
ipmi0: Establishing power cycle handler
random: unblocking device.
arc4random: no preloaded entropy cache
Trying to mount root from zfs:boot-pool/ROOT/default []...
Root mount waiting for: CAM usbus0
Root mount waiting for: CAM usbus0
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x0009> enclosureHandle<0x0001> slot 3
mpr0: At enclosure level 0 and connector name (    )
uhub0: 22 ports with 22 removable, self powered
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x000a> enclosureHandle<0x0001> slot 2
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x000b> enclosureHandle<0x0001> slot 0
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x000c> enclosureHandle<0x0001> slot 1
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x000d> enclosureHandle<0x0001> slot 7
mpr0: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 2
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000d> enclosureHandle<0x0002> slot 3
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000e> enclosureHandle<0x0002> slot 4
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000f> enclosureHandle<0x0002> slot 5
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0010> enclosureHandle<0x0002> slot 6
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0011> enclosureHandle<0x0002> slot 7
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0012> enclosureHandle<0x0002> slot 8
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0013> enclosureHandle<0x0002> slot 9
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0014> enclosureHandle<0x0002> slot 10
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0015> enclosureHandle<0x0002> slot 11
mpr1: At enclosure level 0 and connector name (    )
mpr1: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x0016> enclosureHandle<0x0002> slot 12
mpr1: At enclosure level 0 and connector name (    )
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x000e> enclosureHandle<0x0001> slot 6
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x000f> enclosureHandle<0x0001> slot 4
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <881<SataDev,Direct>,End Device> <6.0Gbps> handle<0x0010> enclosureHandle<0x0001> slot 5
mpr0: At enclosure level 0 and connector name (    )
Root mount waiting for: CAM usbus0
ugen0.2: <vendor 0x0557 product 0x7000> at usbus0
uhub1 on uhub0
uhub1: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 1> on usbus0
uhub1: 4 ports with 3 removable, self powered
Root mount waiting for: CAM usbus0
ugen0.3: <vendor 0x0557 product 0x2419> at usbus0
ukbd0 on uhub1
ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 2> on usbus0
kbd2 at ukbd0
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
ses0 at mpr1 bus 0 scbus1 target 20 lun 0
ses0: <LSI SAS3x28 0601> Fixed Enclosure Services SPC-3 SCSI device
ses0: 1200.000MB/s transfers
ses0: Command Queueing enabled
ses0: SES Device
ses1 at ahciem0 bus 0 scbus8 target 0 lun 0
ses1: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses1: SEMB SES Device
ada0 at ahcich1 bus 0 scbus3 target 0 lun 0
ada0: <TS128GSSD370S P1225CH1> ACS-2 ATA SATA 3.x device
ada0: Serial Number E862862409
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes)
ada0: Command Queueing enabled
ada0: 122104MB (250069680 512 byte sectors)
ada1 at ahcich2 bus 0 scbus4 target 0 lun 0
ada1: <TS120GSSD220S S0628A0> ACS-2 ATA SATA 3.x device
ada1: Serial Number F853320329
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada1: Command Queueing enabled
ada1: 114473MB (234441648 512 byte sectors)
ses1: ada0 in 'Slot 01', SATA Slot: scbus3 target 0
ses1: ada1 in 'Slot 02', SATA Slot: scbus4 target 0
ada2 at ahcich3 bus 0 scbus5 target 0 lun 0
ada2: <TS120GSSD220S S0628A0> ACS-2 ATA SATA 3.x device
ada2: Serial Number F853320393
ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada2: Command Queueing enabled
ada2: 114473MB (234441648 512 byte sectors)
ses1: ada2 in 'Slot 03', SATA Slot: scbus5 target 0
ada3 at ahcich6 bus 0 scbus9 target 0 lun 0
ada3: <TS120GSSD220S S0628A0> ACS-2 ATA SATA 3.x device
ada3: Serial Number F853320399
ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada3: Command Queueing enabled
ada3: 114473MB (234441648 512 byte sectors)
ada4 at ahcich7 bus 0 scbus10 target 0 lun 0
ada4: <TS120GSSD220S S0628A0> ACS-2 ATA SATA 3.x device
ada4: Serial Number F853320398
ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada4: Command Queueing enabled
ada4: 114473MB (234441648 512 byte sectors)
ses0: da8,pass8 in 'Slot00', SAS Slot: 1 phys at slot 0
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db80
ses0: da9,pass9 in 'Slot01', SAS Slot: 1 phys at slot 1
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db81
ses0: da10,pass10 in 'Slot02', SAS Slot: 1 phys at slot 2
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db82
ses0: da11,pass11 in 'Slot03', SAS Slot: 1 phys at slot 3
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db83
ses0: da12,pass12 in 'Slot04', SAS Slot: 1 phys at slot 4
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db84
ses0: da13,pass13 in 'Slot05', SAS Slot: 1 phys at slot 5
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db85
ses0: da14,pass14 in 'Slot06', SAS Slot: 1 phys at slot 6
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db86
ses0: da15,pass15 in 'Slot07', SAS Slot: 1 phys at slot 7
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db87
ses0: da16,pass16 in 'Slot08', SAS Slot: 1 phys at slot 8
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db88
ses0: da17,pass17 in 'Slot09', SAS Slot: 1 phys at slot 9
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db89
ses0: da18,pass18 in 'Slot10', SAS Slot: 1 phys at slot 10
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db8a
ses0: da19,pass19 in 'Slot11', SAS Slot: 1 phys at slot 11
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304801ef3dbbf addr 500304801ef3db8b
da17 at mpr1 bus 0 scbus1 target 17 lun 0
da17: <ATA TS32GSSD370S 2B> Fixed Direct Access SPC-4 SCSI device
da17: Serial Number F758460614
da17: 1200.000MB/s transfers
da17: Command Queueing enabled
da17: 30533MB (62533296 512 byte sectors)
da8 at mpr1 bus 0 scbus1 target 8 lun 0
da8: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da8: Serial Number C160930075
da8: 1200.000MB/s transfers
da8: Command Queueing enabled
da8: 30533MB (62533296 512 byte sectors)
da1 at mpr0 bus 0 scbus0 target 1 lun 0
da1: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number C571900873
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 61057MB (125045424 512 byte sectors)
da0 at mpr0 bus 0 scbus0 target 0 lun 0
da0: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number C571900869
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 61057MB (125045424 512 byte sectors)
da3 at mpr0 bus 0 scbus0 target 3 lun 0
da3: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number C571900885
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 61057MB (125045424 512 byte sectors)
da2 at mpr0 bus 0 scbus0 target 2 lun 0
da2: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number C571900876
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 61057MB (125045424 512 byte sectors)
da10 at mpr1 bus 0 scbus1 target 10 lun 0
da10: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da10: Serial Number C364420142
da10: 1200.000MB/s transfers
da10: Command Queueing enabled
da10: 30533MB (62533296 512 byte sectors)
da6 at mpr0 bus 0 scbus0 target 6 lun 0
da6: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number C571900866
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 61057MB (125045424 512 byte sectors)
da4 at mpr0 bus 0 scbus0 target 4 lun 0
da4: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number C571900855
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 61057MB (125045424 512 byte sectors)
da5 at mpr0 bus 0 scbus0 target 5 lun 0
da5: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number C571900865
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 61057MB (125045424 512 byte sectors)
da11 at mpr1 bus 0 scbus1 target 11 lun 0
da11: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da11: Serial Number C364420148
da11: 1200.000MB/s transfers
da11: Command Queueing enabled
da11: 30533MB (62533296 512 byte sectors)
da12 at mpr1 bus 0 scbus1 target 12 lun 0
da12: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da12: Serial Number C333750271
da12: 1200.000MB/s transfers
da12: Command Queueing enabled
da12: 30533MB (62533296 512 byte sectors)
da7 at mpr0 bus 0 scbus0 target 7 lun 0
da7: <ATA TS64GSSD370S 4D> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number C571900867
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 61057MB (125045424 512 byte sectors)
da13 at mpr1 bus 0 scbus1 target 13 lun 0
da13: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da13: Serial Number C392050392
da13: 1200.000MB/s transfers
da13: Command Queueing enabled
da13: 30533MB (62533296 512 byte sectors)
da18 at mpr1 bus 0 scbus1 target 18 lun 0
da18: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da18: Serial Number C160931700
da18: 1200.000MB/s transfers
da18: Command Queueing enabled
da18: 30533MB (62533296 512 byte sectors)
da14 at mpr1 bus 0 scbus1 target 14 lun 0
da14: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da14: Serial Number C392050400
da14: 1200.000MB/s transfers
da14: Command Queueing enabled
da14: 30533MB (62533296 512 byte sectors)
da9 at mpr1 bus 0 scbus1 target 9 lun 0
da9: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da9: Serial Number C364420131
da9: 1200.000MB/s transfers
da9: Command Queueing enabled
da9: 30533MB (62533296 512 byte sectors)
da16 at mpr1 bus 0 scbus1 target 16 lun 0
da16: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da16: Serial Number C364420439
da16: 1200.000MB/s transfers
da16: Command Queueing enabled
da16: 30533MB (62533296 512 byte sectors)
da15 at mpr1 bus 0 scbus1 target 15 lun 0
da15: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da15: Serial Number C364420430
da15: 1200.000MB/s transfers
da15: Command Queueing enabled
da15: 30533MB (62533296 512 byte sectors)
da19 at mpr1 bus 0 scbus1 target 19 lun 0
da19: <ATA TS32GSSD370S 4H> Fixed Direct Access SPC-4 SCSI device
da19: Serial Number C304332037
da19: 1200.000MB/s transfers
da19: Command Queueing enabled
da19: 30533MB (62533296 512 byte sectors)
ums0 on uhub1
ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 2> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
lo0: link state changed to UP
igb0: link state changed to UP
GEOM_ELI: Device ada0p3.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: hardware
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI>
igb0: link state changed to DOWN
igb0: link state changed to UP
CPU: Intel(R) Pentium(R) CPU G4560 @ 3.50GHz (3504.12-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906e9  Family=0x6  Model=0x9e  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4ffaebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x2946687<FSGSBASE,TSCADJ,SGX,SMEP,ERMS,INVPCID,NFPUSG,MPX,RDSEED,SMAP,CLFLUSHOPT,PROCTRACE>
  Structured Extended Features3=0x9c002400<MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
Security policy loaded: MAC/ntpd (mac_ntpd)
pid 2174 (syslog-ng), jid 0, uid 0: exited on signal 6 (core dumped)

It looks like you are running an older firmware 16.00.01.00. A lot of bugs have been fixed in 16.00.10.00 since that version. But I can try it, just to see if that's the problem. Thanks for getting back with info

Johnnie Black · Oct 15, 2020

Lars Jensen said:
A lot of bugs have been fixed in 16.00.10.00 since that version.

I can also try again after updating, like mentioned these are on a test server, so didn't have the need to update for now.

Johnnie Black · Oct 15, 2020

Update firmware to 16.00.10.00, still all good.

Code:

root@truenas[~]# sas3flash -listall
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

        Adapter Selected is a Avago SAS: SAS3008(C0)

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS3008(C0)  16.00.10.00    0e.01.00.07    08.37.00.00     00:01:00:00
1  SAS3008(C0)  16.00.10.00    0e.01.00.07    08.37.00.00     00:02:00:00

        Finished Processing Commands Successfully.
        Exiting SAS3Flash.

Lars Jensen · Oct 20, 2020

Thanks for testing!

I now have no errors with both cables and the latest firmware .10/.12. Very odd!

Tested with SAS disks. It's like when after I connected SAS disks one time it works, even after I disconnect them.

Attached an image of the backplane connections (haven't changed those, they are still in place like before). The server was originally assembled by others, and I guess the connection between HBA and backplane might have had an issue.

Important Announcement for The TrueNAS Community.

SAS 9300-8i dual linked to BPN-SAS3-826EL1 = I/O errors on all SATA SSDs ?

Lars Jensen

Explorer

Attachments

sretalla

Powered by Neutrality

LSI 9300-xx Firmware Update

Lars Jensen

Explorer

sretalla

Powered by Neutrality

Johnnie Black

Guru

Lars Jensen

Explorer

Johnnie Black

Guru

Lars Jensen

Explorer

Johnnie Black

Guru

Lars Jensen

Explorer

Johnnie Black

Guru

Lars Jensen

Explorer

Attachments

Johnnie Black

Guru

Johnnie Black

Guru

Lars Jensen

Explorer

Attachments

Similar threads

Important Announcement for The TrueNAS Community.