Truenas only showing one of two disk arrays

S

Super7800

Dabbler
Joined
Feb 2, 2021
Messages
16
i have a HP DL380 g7. My setup specs are as follows:
-1x E5504 CPU
-72 gb ram (8gb sticks)
-1x dell h200 lsi 9210-8i (This handles 4x 146gb sas HDDs for boot disk. Uses backplane of DL380 server chassis)
-2x LSI SFF-8088 cards (one for each disk shelf, using two cables each)
-2x Symantec promise Jbod disk arrays (each with 16x 3TB HDDs)
-1x intel i520 SFP+ Network card

So truenas is only seeing one of the two disk shelfs. During install setup it sees both, and the LSI configuration sees both. In truenas storage > disks it even starts the numbering off at da16, but is missing da0 to da15 (one entire disk array). My boot disk is 4x sas 146gb drives in a mirrored array. The lsi card should automatically failover in case of a drive failure, atleast in theory. All cards are flashed to IT mode. No hardware raid.

the two LSI SFF-8088 show up as only one card in the LSI configurator, and in bios again only show up as one device (both shown as just one pci device and one boot device). This is probably normal as the LSI config does show both disk shelfs but only one card.

I have reinstalled Truenas twice. no luck. I am by far not an expert. Any suggestions? im sure i left an important detail out, so please let me know any further information that is needed. Thanks.
 
artlessknave

artlessknave

Wizard
Joined
Oct 29, 2016
Messages
1,506
Super7800 said:
The lsi card should automatically failover in case of a drive failure, atleast in theory.
Click to expand...
HBA's do not handle failover. what are you trying to say here?
Super7800 said:
2x LSI SFF-8088 cards (one for each disk shelf, using two cables each)
Click to expand...
the rest of your hardware you specify but "LSI SFF-8088" is not a part a number, so we have no idea what those are.

the first thing I see would be to check that your "LSI SFF-8088" cards are working correctly (unless they are RAID cards, in which case toss em and get some external HBA's).
 
S

Super7800

Dabbler
Joined
Feb 2, 2021
Messages
16
i have absolutely no idea what im saying here. I do not know how failover works on truenas yet. a youtube video showing how to set it (dell h200 lsi 9210-8i) up said that. The "lsi configurator" (NOT RAID) allowed me to select the other three drives as "failover drives" incase it failed to boot off of the "primary" boot drive.

here is the ebay link to the cards "sff-8088". they are not flashed to raid since half the disks are functioning correctly (at least i assume so?). sff-8088.
here is the link for the other card that controls the 4 boot disks: dell H200

during setup (install), truenas sees all the disks. the "lsi manager" also sees both disk arrays and all disks. I think that means they are working correctly? Below is what happens when i run the "dmesg" command. Maybe thats helpful?

Code:
Last login: Sun Jan  8 17:55:05 on pts/0
FreeBSD 13.1-RELEASE-p2 n245412-484f039b1d0 TRUENAS

        TrueNAS (c) 2009-2022, iXsystems, Inc.
        All rights reserved.
        TrueNAS code is released under the modified BSD license with some
        files copyrighted by (c) iXsystems, Inc.

        For more information, documentation, help or support, go here:
        http://truenas.com
Welcome to TrueNAS

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@truenas[~]# dmesg
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.1-RELEASE-p2 n245412-484f039b1d0 TRUENAS amd64
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU           E5504  @ 2.00GHz (1999.50-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a  Stepping=5
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID
  TSC: P-state invariant, performance statistics
real memory  = 77309411328 (73728 MB)
avail memory = 75199807488 (71716 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <HP     ProLiant>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
Firmware Warning (ACPI): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20201113/tbfadt-850)
Firmware Warning (ACPI): Invalid length for FADT/Pm2ControlBlock: 32, using default 8 (20201113/tbfadt-850)
ioapic1 <Version 2.0> irqs 24-47
ioapic0 <Version 2.0> irqs 0-23
Launching APs: 3 1 2
random: entropy device external interface
kbd1 at kbdmux0
vtvga0: <VT VGA driver>
aesni0: No AES or SHA support.
padlock0: No ACE support.
acpi0: <HP ProLiant>
acpi0: Power Button (fixed)
acpi0: _OSC failed: AE_BUFFER_OVERFLOW
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
apei0: <ACPI Platform Error Interface> on acpi0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci5: <ACPI PCI bus> on pcib5
mps0: <Avago Technologies (LSI) SAS2008> port 0x6000-0x60ff mem 0xfbff0000-0xfbff3fff,0xfbf80000-0xfbfbffff irq 26 at device 0.0 on pci5
mps0: Firmware: 13.00.57.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib6: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci7: <ACPI PCI bus> on pcib7
mps1: <Avago Technologies (LSI) SAS2008> port 0x5000-0x50ff mem 0xfbef0000-0xfbefffff,0xfbe80000-0xfbebffff irq 30 at device 0.0 on pci7
mps1: Firmware: 07.15.08.00, Driver: 21.02.00.00-fbsd
mps1: IOCCapabilities: 185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR>
pcib8: <ACPI PCI-PCI bridge> at device 8.0 on pci0
pci8: <ACPI PCI bus> on pcib8
ix0: <Intel(R) X520 82599ES (SFI/SFP+)> port 0x4000-0x401f mem 0xfbd80000-0xfbdfffff,0xfbd70000-0xfbd73fff irq 31 at device 0.0 on pci8
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 4 RX queues 4 TX queues
ix0: Using MSI-X interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: 90:e2:ba:30:6c:78
ix0: PCI Express Bus: Speed 5.0GT/s Width x4
ix0: eTrack 0x0001546c PHY FW V65535
ix1: <Intel(R) X520 82599ES (SFI/SFP+)> port 0x4020-0x403f mem 0xfbc80000-0xfbcfffff,0xfbc70000-0xfbc73fff irq 39 at device 0.1 on pci8
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 4 RX queues 4 TX queues
ix1: Using MSI-X interrupts with 5 vectors
ix1: allocated for 4 queues
ix1: allocated for 4 rx queues
ix1: Ethernet address: 90:e2:ba:30:6c:79
ix1: PCI Express Bus: Speed 5.0GT/s Width x4
ix1: eTrack 0x0001546c PHY FW V65535
pcib9: <ACPI PCI-PCI bridge> at device 9.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 10.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached)
pcib11: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci11: <ACPI PCI bus> on pcib11
bce0: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci11
miibus0: <MII bus> on bce0
brgphy0: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce0: Using defaults for TSO: 65518/35/2048
bce0: Ethernet address: 80:c1:6e:23:a2:46
bce0: ASIC (0x57092003);
bce0: link state changed to DOWN
Rev (C0); Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12)
Coal (RX:6,6,18,18; TX:20,20,80,80)
bce1: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf2000000-0xf3ffffff irq 17 at device 0.1 on pci11
miibus1: <MII bus> on bce1
brgphy1: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce1: Using defaults for TSO: 65518/35/2048
bce1: Ethernet address: 80:c1:6e:23:a2:48
bce1: ASIC (0x57092003);
bce1: link state changed to DOWN
Rev (C0); Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib12: <ACPI PCI-PCI bridge> at device 28.2 on pci0
pci12: <ACPI PCI bus> on pcib12
bce2: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf8000000-0xf9ffffff irq 18 at device 0.0 on pci12
miibus2: <MII bus> on bce2
brgphy2: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus2
brgphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce2: Using defaults for TSO: 65518/35/2048
bce2: Ethernet address: 80:c1:6e:23:a2:4a
bce2: ASIC (0x57092003); Rev (C0); Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI)
Coal (RX:6,6,18,18; TX:20,20,80,80)
bce3: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf6000000-0xf7ffffff irq 19 at device 0.1 on pci12
miibus3: <MII bus> on bce3
brgphy3: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus3
brgphy3:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce3: Using defaults for TSO: 65518/35/2048
bce3: Ethernet address: 80:c1:6e:23:a2:4c
bce3: ASIC (0x57092003); Rev (C0); Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib13: <ACPI PCI-PCI bridge> at device 28.4 on pci0
pci13: <ACPI PCI bus> on pcib13
uhci0: <HP iLO Standard Virtual USB controller> port 0x3c00-0x3c1f irq 17 at device 0.4 on pci13
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1: <Intel 82801JI (ICH10) USB controller USB-A> port 0x1000-0x101f irq 20 at device 29.0 on pci0
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2: <Intel 82801JI (ICH10) USB controller USB-B> port 0x1020-0x103f irq 23 at device 29.1 on pci0
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
uhci3: <Intel 82801JI (ICH10) USB controller USB-C> port 0x1040-0x105f irq 22 at device 29.2 on pci0
usbus3 on uhci3
usbus3: 12Mbps Full Speed USB v1.0
uhci4: <Intel 82801JI (ICH10) USB controller USB-F> port 0x1060-0x107f irq 23 at device 29.3 on pci0
usbus4 on uhci4
usbus4: 12Mbps Full Speed USB v1.0
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0xf1bf0000-0xf1bf03ff irq 20 at device 29.7 on pci0
usbus5: EHCI version 1.0
usbus5 on ehci0
usbus5: 480Mbps High Speed USB v2.0
pcib14: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci14: <ACPI PCI bus> on pcib14
vgapci0: <VGA-compatible display> port 0x2000-0x20ff mem 0xe8000000-0xefffffff,0xf1cf0000-0xf1cfffff irq 23 at device 3.0 on pci14
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_tz0: <Thermal Zone> on acpi0
ipmi0: <IPMI System Interface> port 0xca2-0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0.
psm0: model IntelliMouse Explorer, device ID 4
acpi_syscontainer0: <System Container> port 0x2e-0x2f,0x620-0x65f,0x680-0x69f,0x600-0x61f,0x660-0x67f,0x300-0x31f on acpi0
uart0: <Non-standard ns8250 class UART with FIFOs> port 0x3f8-0x3ff irq 4 flags0x10 on acpi0
ichwd0: <Intel ICH10 watchdog timer> on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
ichwd0: <Intel ICH10 watchdog timer> at port 0x930-0x937,0x960-0x97f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff pnpid ORM0000 on isa0
uart1: <Non-standard ns8250 class UART with FIFOs> at port 0x2f8 irq 3 on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr f
device_attach: est0 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr f
device_attach: est1 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr f
device_attach: est2 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr f
device_attach: est3 attach returned 6
Timecounter "TSC" frequency 1999431679 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
ugen0.1: <HP UHCI root HUB> at usbus0
ugen2.1: <Intel UHCI root HUB> at usbus2
ugen3.1: <Intel UHCI root HUB> at usbus3
ugen1.1: <Intel UHCI root HUB> at usbus1
uhub0 on usbus0
uhub0: <HP UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub1 on usbus3
uhub2 on usbus2
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen4.1: <Intel UHCI root HUB> at usbus4
ugen5.1: <Intel EHCI root HUB> at usbus5
uhub3 on usbus1
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
uhub4 on usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
uhub5 on usbus5
uhub5: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
ipmi0: IPMI device rev. 1, firmware rev. 1.89, version 2.0, device support mask0x8f
ipmi0: Number of channels 1
ipmi0: Attached watchdog
ipmi0: Establishing power cycle handler
Trying to mount root from zfs:boot-pool/ROOT/default []...
uhub1: 2 ports with 2 removable, self powered
uhub0: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
ugen0.2: <HP Virtual Keyboard> at usbus0
ukbd0 on uhub0
ukbd0: <Virtual Keyboard > on usbus0
kbd2 at ukbd0
ums0 on uhub0
ums0: <Virtual Mouse> on usbus0
Root mount waiting for: CAM usbus5
Root mount waiting for: CAM usbus5
uhub5: 8 ports with 8 removable, self powered
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
da0 at mps0 bus 0 scbus0 target 8 lun 0
da0: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number YVJSDU3D
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 2861588MB (5860533168 512 byte sectors)
da1 at mps0 bus 0 scbus0 target 9 lun 0
da1: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number YVKMX11K
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 2861588MB (5860533168 512 byte sectors)
da2 at mps0 bus 0 scbus0 target 10 lun 0
da2: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number YVKP6T9K
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 2861588MB (5860533168 512 byte sectors)
da3 at mps0 bus 0 scbus0 target 11 lun 0
da3: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number YVKUA2WK
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 2861588MB (5860533168 512 byte sectors)
da4 at mps0 bus 0 scbus0 target 12 lun 0
da4: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number YVKNEURK
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 2861588MB (5860533168 512 byte sectors)
da7 at mps0 bus 0 scbus0 target 15 lun 0
da7: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number YVKTYMZK
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 2861588MB (5860533168 512 byte sectors)
da9 at mps0 bus 0 scbus0 target 17 lun 0
da9: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da9: Serial Number YVJT1HBK
da9: 600.000MB/s transfers
da9: Command Queueing enabled
da9: 2861588MB (5860533168 512 byte sectors)
da11 at mps0 bus 0 scbus0 target 19 lun 0
da11: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da11: Serial Number YVKP5EAK
da11: 600.000MB/s transfers
da11: Command Queueing enabled
da11: 2861588MB (5860533168 512 byte sectors)
da10 at mps0 bus 0 scbus0 target 18 lun 0
da10: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da10: Serial Number YVKVU0KK
da10: 600.000MB/s transfers
da10: Command Queueing enabled
da10: 2861588MB (5860533168 512 byte sectors)
da13 at mps0 bus 0 scbus0 target 21 lun 0
da13: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da13: Serial Number YVJJ3LMD
da13: 600.000MB/s transfers
da13: Command Queueing enabled
da13: 2861588MB (5860533168 512 byte sectors)
da14 at mps0 bus 0 scbus0 target 22 lun 0
da14: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da14: Serial Number YVKP6UGK
da14: 600.000MB/s transfers
da14: Command Queueing enabled
da14: 2861588MB (5860533168 512 byte sectors)
da12 at mps0 bus 0 scbus0 target 20 lun 0
da12: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da12: Serial Number P9HMRN0W
da12: 600.000MB/s transfers
da12: Command Queueing enabled
da12: 2861588MB (5860533168 512 byte sectors)
da15 at mps0 bus 0 scbus0 target 23 lun 0
da15: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da15: Serial Number YVKTY7GK
da15: 600.000MB/s transfers
da15: Command Queueing enabled
da15: 2861588MB (5860533168 512 byte sectors)
da18 at mps0 bus 0 scbus0 target 47 lun 0
da18: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da18: Serial Number YVKP6T9K
da18: 600.000MB/s transfers
da18: Command Queueing enabled
da18: 2861588MB (5860533168 512 byte sectors)
da17 at mps0 bus 0 scbus0 target 46 lun 0
da17: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da17: Serial Number YVKMX11K
da17: 600.000MB/s transfers
da17: Command Queueing enabled
da17: 2861588MB (5860533168 512 byte sectors)
da19 at mps0 bus 0 scbus0 target 48 lun 0
da19: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da19: Serial Number YVKUA2WK
da19: 600.000MB/s transfers
da19: Command Queueing enabled
da19: 2861588MB (5860533168 512 byte sectors)
da16 at mps0 bus 0 scbus0 target 45 lun 0
da16: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da16: Serial Number YVJSDU3D
da16: 600.000MB/s transfers
da16: Command Queueing enabled
da16: 2861588MB (5860533168 512 byte sectors)
da20 at mps0 bus 0 scbus0 target 49 lun 0
da20: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da20: Serial Number YVKNEURK
da20: 600.000MB/s transfers
da20: Command Queueing enabled
da20: 2861588MB (5860533168 512 byte sectors)
da6 at mps0 bus 0 scbus0 target 14 lun 0
da6: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number P9HN7B4W
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 2861588MB (5860533168 512 byte sectors)
da5 at mps0 bus 0 scbus0 target 13 lun 0
da5: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number P9G7PRRW
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 2861588MB (5860533168 512 byte sectors)
da8 at mps0 bus 0 scbus0 target 16 lun 0
da8: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da8: Serial Number P9J262LW
da8: 600.000MB/s transfers
da8: Command Queueing enabled
da8: 2861588MB (5860533168 512 byte sectors)
ses0 at mps0 bus 0 scbus0 target 44 lun 0
ses0: <Promise J630s 060=> Fixed Enclosure Services SPC-4 SCSI device
ses0: Serial Number TB0038052886
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SES Device
da23 at mps0 bus 0 scbus0 target 52 lun 0
da23: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da23: Serial Number YVKTYMZK
da23: 600.000MB/s transfers
da23: Command Queueing enabled
da23: 2861588MB (5860533168 512 byte sectors)
da25 at mps0 bus 0 scbus0 target 54 lun 0
da25: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da25: Serial Number YVJT1HBK
da25: 600.000MB/s transfers
da25: Command Queueing enabled
da25: 2861588MB (5860533168 512 byte sectors)
ses1 at mps0 bus 0 scbus0 target 81 lun 0
ses1: <Promise J630s 060=> Fixed Enclosure Services SPC-4 SCSI device
ses1: Serial Number TB0038052886
ses1: 600.000MB/s transfers
ses1: Command Queueing enabled
ses1: SES Device
da22 at mps0 bus 0 scbus0 target 51 lun 0
da22: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da22: Serial Number P9HN7B4W
da22: 600.000MB/s transfers
da22: Command Queueing enabled
da22: 2861588MB (5860533168 512 byte sectors)
da21 at mps0 bus 0 scbus0 target 50 lun 0
da21: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da21: Serial Number P9G7PRRW
da21: 600.000MB/s transfers
da21: Command Queueing enabled
da21: 2861588MB (5860533168 512 byte sectors)
da27 at mps0 bus 0 scbus0 target 56 lun 0
da27: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da27: Serial Number YVKP5EAK
da27: 600.000MB/s transfers
da27: Command Queueing enabled
da27: 2861588MB (5860533168 512 byte sectors)
da26 at mps0 bus 0 scbus0 target 55 lun 0
da26: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da26: Serial Number YVKVU0KK
da26: 600.000MB/s transfers
da26: Command Queueing enabled
da26: 2861588MB (5860533168 512 byte sectors)
da24 at mps0 bus 0 scbus0 target 53 lun 0
da24: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da24: Serial Number P9J262LW
da24: 600.000MB/s transfers
da24: Command Queueing enabled
da24: 2861588MB (5860533168 512 byte sectors)
da31 at mps0 bus 0 scbus0 target 60 lun 0
da31: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da31: Serial Number YVKTY7GK
da31: 600.000MB/s transfers
da31: Command Queueing enabled
da31: 2861588MB (5860533168 512 byte sectors)
da29 at mps0 bus 0 scbus0 target 58 lun 0
da29: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da29: Serial Number YVJJ3LMD
da29: 600.000MB/s transfers
da29: Command Queueing enabled
da29: 2861588MB (5860533168 512 byte sectors)
da30 at mps0 bus 0 scbus0 target 59 lun 0
da30: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da30: Serial Number YVKP6UGK
da30: 600.000MB/s transfers
da30: Command Queueing enabled
da30: 2861588MB (5860533168 512 byte sectors)
da28 at mps0 bus 0 scbus0 target 57 lun 0
da28: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da28: Serial Number P9HMRN0W
da28: 600.000MB/s transfers
da28: Command Queueing enabled
da28: 2861588MB (5860533168 512 byte sectors)
da34 at mps1 bus 0 scbus1 target 9 lun 0
da34: <HP DG0146FAMWL HPDC> Fixed Direct Access SPC-3 SCSI device
da34: Serial Number 3SD13EP400009005J5LK
da34: 600.000MB/s transfers
da34: Command Queueing enabled
da34: 140014MB (286749488 512 byte sectors)
da35 at mps1 bus 0 scbus1 target 10 lun 0
da35: <HP DG0146FAMWL HPDB> Fixed Direct Access SPC-3 SCSI device
da35: Serial Number 3SD0QESR00009953RW2U
da35: 600.000MB/s transfers
da35: Command Queueing enabled
da35: 140014MB (286749488 512 byte sectors)
da32 at mps1 bus 0 scbus1 target 7 lun 0
da32: <HP DG146ABAB4 HPDC> Fixed Direct Access SPC-3 SCSI device
da32: Serial Number 3NM4BY200000982745Y8
da32: 300.000MB/s transfers
da32: Command Queueing enabled
da32: 140014MB (286749488 512 byte sectors)
da33 at mps1 bus 0 scbus1 target 8 lun 0
da33: <HP DG146BB976 HPDC> Fixed Direct Access SPC-3 SCSI device
da33: Serial Number 3NM5606F000098388V18
da33: 300.000MB/s transfers
da33: Command Queueing enabled
da33: 140014MB (286749488 512 byte sectors)
ses0: da0,pass0 in Array Device Slot 1, SAS Slot: 1+ phys at slot 1
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03e9b4e86
ses0: da1,pass1 in Array Device Slot 2, SAS Slot: 1+ phys at slot 2
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecd4b32
ses1: da16,pass17 in Array Device Slot 1, SAS Slot: 1+ phys at slot 1
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03e9b4e85
ses0: da2,pass2 in Array Device Slot 3, SAS Slot: 1+ phys at slot 3
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecfaf16
ses1: da17,pass18 in Array Device Slot 2, SAS Slot: 1+ phys at slot 2
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecd4b31
ses0: da3,pass3 in Array Device Slot 4, SAS Slot: 1+ phys at slot 4
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed726b6
ses1: da18,pass19 in Array Device Slot 3, SAS Slot: 1+ phys at slot 3
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecfaf15
ses0: da4,pass4 in Array Device Slot 5, SAS Slot: 1+ phys at slot 5
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ece4766
ses1: da19,pass20 in Array Device Slot 4, SAS Slot: 1+ phys at slot 4
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed726b5
ses0: da5,pass5 in Array Device Slot 6, SAS Slot: 1+ phys at slot 6
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca0580e00ba
ses1: da20,pass21 in Array Device Slot 5, SAS Slot: 1+ phys at slot 5
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ece4765
ses0: da6,pass6 in Array Device Slot 7, SAS Slot: 1+ phys at slot 7
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca0585d2a36
ses1: da21,pass22 in Array Device Slot 6, SAS Slot: 1+ phys at slot 6
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca0580e00b9
ses0: da7,pass7 in Array Device Slot 8, SAS Slot: 1+ phys at slot 8
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed67ace
ses1: da22,pass23 in Array Device Slot 7, SAS Slot: 1+ phys at slot 7
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca0585d2a35
ses0: da8,pass8 in Array Device Slot 9, SAS Slot: 1+ phys at slot 9
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca05874ba5a
ses1: da23,pass24 in Array Device Slot 8, SAS Slot: 1+ phys at slot 8
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed67acd
ses0: da9,pass9 in Array Device Slot 10, SAS Slot: 1+ phys at slot 10
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03e9c761e
ses1: da24,pass25 in Array Device Slot 9, SAS Slot: 1+ phys at slot 9
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca05874ba59
ses0: da10,pass10 in Array Device Slot 11, SAS Slot: 1+ phys at slot 11
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed9d84e
ses1: da25,pass26 in Array Device Slot 10, SAS Slot: 1+ phys at slot 10
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03e9c761d
ses0: da11,pass11 in Array Device Slot 12, SAS Slot: 1+ phys at slot 12
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecf9ac2
ses1: da26,pass27 in Array Device Slot 11, SAS Slot: 1+ phys at slot 11
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed9d84d
ses0: da12,pass12 in Array Device Slot 13, SAS Slot: 1+ phys at slot 13
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca0585c3e42
ses1: da27,pass28 in Array Device Slot 12, SAS Slot: 1+ phys at slot 12
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecf9ac1
ses0: da13,pass13 in Array Device Slot 14, SAS Slot: 1+ phys at slot 14
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03e8e09da
ses1: da28,pass29 in Array Device Slot 13, SAS Slot: 1+ phys at slot 13
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca0585c3e41
ses0: da14,pass14 in Array Device Slot 15, SAS Slot: 1+ phys at slot 15
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecfafa6
ses1: da29,pass30 in Array Device Slot 14, SAS Slot: 1+ phys at slot 14
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03e8e09d9
ses0: da15,pass15 in Array Device Slot 16, SAS Slot: 1+ phys at slot 16
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed674be
ses1: da30,pass31 in Array Device Slot 15, SAS Slot: 1+ phys at slot 15
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecfafa5
ses1: da31,pass32 in Array Device Slot 16, SAS Slot: 1+ phys at slot 16
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed674bd
lo0: link state changed to UP
GEOM_MULTIPATH: disk13 created
GEOM_MULTIPATH: da28 added to disk13
GEOM_MULTIPATH: da28 is now active path in disk13
GEOM_MULTIPATH: disk15 created
GEOM_MULTIPATH: da30 added to disk15
GEOM_MULTIPATH: da30 is now active path in disk15
GEOM_MULTIPATH: disk14 created
GEOM_MULTIPATH: da29 added to disk14
GEOM_MULTIPATH: da29 is now active path in disk14
GEOM_MULTIPATH: disk16 created
GEOM_MULTIPATH: da31 added to disk16
GEOM_MULTIPATH: da31 is now active path in disk16
GEOM_MULTIPATH: disk9 created
GEOM_MULTIPATH: da24 added to disk9
GEOM_MULTIPATH: da24 is now active path in disk9
GEOM_MULTIPATH: disk11 created
GEOM_MULTIPATH: da26 added to disk11
GEOM_MULTIPATH: da26 is now active path in disk11
GEOM_MULTIPATH: disk6 created
GEOM_MULTIPATH: da21 added to disk6
GEOM_MULTIPATH: da21 is now active path in disk6
GEOM_MULTIPATH: disk12 created
GEOM_MULTIPATH: da27 added to disk12
GEOM_MULTIPATH: da27 is now active path in disk12
GEOM_MULTIPATH: disk7 created
GEOM_MULTIPATH: da22 added to disk7
GEOM_MULTIPATH: da22 is now active path in disk7
GEOM_MULTIPATH: disk10 created
GEOM_MULTIPATH: da25 added to disk10
GEOM_MULTIPATH: da25 is now active path in disk10
GEOM_MULTIPATH: disk8 created
GEOM_MULTIPATH: da23 added to disk8
GEOM_MULTIPATH: da23 is now active path in disk8
GEOM_MULTIPATH: da8 added to disk9
GEOM_MULTIPATH: da5 added to disk6
GEOM_MULTIPATH: da6 added to disk7
GEOM_MULTIPATH: disk5 created
GEOM_MULTIPATH: da20 added to disk5
GEOM_MULTIPATH: da20 is now active path in disk5
GEOM_MULTIPATH: disk1 created
GEOM_MULTIPATH: da16 added to disk1
GEOM_MULTIPATH: da16 is now active path in disk1
GEOM_MULTIPATH: disk4 created
GEOM_MULTIPATH: da19 added to disk4
GEOM_MULTIPATH: da19 is now active path in disk4
GEOM_MULTIPATH: disk2 created
GEOM_MULTIPATH: da17 added to disk2
GEOM_MULTIPATH: da17 is now active path in disk2
GEOM_MULTIPATH: disk3 created
GEOM_MULTIPATH: da18 added to disk3
GEOM_MULTIPATH: da18 is now active path in disk3
GEOM_MULTIPATH: da15 added to disk16
GEOM_MULTIPATH: da12 added to disk13
GEOM_MULTIPATH: da14 added to disk15
GEOM_MULTIPATH: da13 added to disk14
GEOM_MULTIPATH: da10 added to disk11
GEOM_MULTIPATH: da11 added to disk12
GEOM_MULTIPATH: da9 added to disk10
GEOM_MULTIPATH: da7 added to disk8
GEOM_MULTIPATH: da4 added to disk5
GEOM_MULTIPATH: da3 added to disk4
GEOM_MULTIPATH: da2 added to disk3
GEOM_MULTIPATH: da1 added to disk2
GEOM_MULTIPATH: da0 added to disk1
CPU: Intel(R) Xeon(R) CPU           E5504  @ 2.00GHz (1999.43-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a  Stepping=5
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features3=0x9c000000<IBPB,STIBP,L1DFL,SSBD>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID
  TSC: P-state invariant, performance statistics
bce2: link state changed to DOWN
bce3: link st
 
Last edited:
Davvo

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
Super7800 said:
-1x dell h200 lsi 9210-8i (This handles 4x 146gb sas HDDs for boot disk.
Click to expand...
TrueNAS strongly suggests users to use SSDs for the boot-pool. Strongly.
It's not related with your current issue though.
 
S

Super7800

Dabbler
Joined
Feb 2, 2021
Messages
16
TrueNAS strongly suggests users to use SSDs for the boot-pool. Strongly.
Click to expand...
ok. That will definitely be something to look into. I figured the 10K SAS drives would have the speed, but perhaps not.

i ran the sesutil status and sesutil map commands, which seems to show both enclosures. Here is the results of those commands: https://pastebin.com/aQkDMNBi
 
artlessknave

artlessknave

Wizard
Joined
Oct 29, 2016
Messages
1,506
Super7800 said:
The "lsi configurator" (NOT RAID) allowed me to select the other three drives as "failover drives" incase it failed to boot off of the "primary" boot drive.
Click to expand...
this doesn't make sense to me. an HBA does not have "Failover drives". all it does is attach drives to the system, the system does all the failover.
Super7800 said:
"sff-8088"
Click to expand...
SFF-8088 is a connector, not a card. I would never put these cards in a system. the fact that they claim LSI but can't even put the model of the card is extremely suspicious. the contents of the description is tech speak word salad. there should be stickers that at least say which LSI model, but they have only given a picture of the front of the card.

my current theory is that you have fake cards, and the fake cards do all kinds of terrible things. I have had fake cards myself, and I had one of them that would behave similarly. it was present, but it would not present any drives.

the DELL h200 looks far less suspicious and the fakes seem to tend to be Intel or LSI. unfortunately, your connections will not let you test with that one.

@jgreco what does the resident Grinch think?
 
ChrisRJ

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,919
Davvo said:
TrueNAS strongly suggests users to use SSDs for the boot-pool. Strongly.
It's not related with your current issue though.
Click to expand...
In my understanding this is primarily meant to say "do not use USB sticks but SSDs". And with today's prices an SSD is cheap enough that a general recommendation makes sense. But I personally wouldn't read it such that HDDs, like in such a setup here, should be replaced or even cause issues.
 
ChrisRJ

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,919
Super7800 said:
i have absolutely no idea what im saying here. I do not know how failover works on truenas yet. a youtube video showing how to set it (dell h200 lsi 9210-8i) up said that. The "lsi configurator" (NOT RAID) allowed me to select the other three drives as "failover drives" incase it failed to boot off of the "primary" boot drive.
Click to expand...
Can you link this YouTube video here? As a general comment, since I haven't seen the video in question, it must be said that an enormous amount of bullshit is out there on YouTube. Whether or not the relevant video falls into this category remains to be seen.

Fail-over and other HA topics are highly advanced stuff. They all come with specific and not-negotiable prerequisites, edge conditions, etc. Also, operations need to follow the rules that apply. All in all it is very dangerous to do this without knowing these things. Because in fact HA setups, while mitigating the risks they are supposed to, often introduce additional risks, if things are not done properly.

With my current understanding of the situation, I would recommend to change things for a normal setup. The biggest risk for data loss is still human error. And with an HA setup, there is a lot more room for human error than with something simpler. If that is a playground system that will never hold important data, things are different, of course.
 
S

Super7800

Dabbler
Joined
Feb 2, 2021
Messages
16
ChrisRJ said:
Can you link this YouTube video here? As a general comment, since I haven't seen the video in question, it must be said that an enormous amount of bullshit is out there on YouTube. Whether or not the relevant video falls into this category remains to be seen.
Click to expand...
Here is the YouTube video: Videooooo
I also bought another card because I was having issues with the Dell h200 card prior to finding the video. 9211-8i card

I don't think the two other cards are fake, but im not saying they aren't either. They come from a local company that I've done buisness with in the past. They aren't very knowledgeable tho so they could have gotten fake cards and not known it. They advertise everything like technobable keywords Overload. I will take them out tonight and take detailed pictures, perhaps that will shed some light on their authenticity. I'd hate to just keep buying cards when the issue is something else.

This will not be a playground type thing. Once I get it working it won't be touched for years, only if there is hardware failure, so yeah if failover is too complex (which it sounds like) I won't even attempt it. I'll just keep the 4 drives in mirrored anyways (all my hp servers have 4 drives, and I have no other use for them).

Thanks for the input and help!
 
artlessknave

artlessknave

Wizard
Joined
Oct 29, 2016
Messages
1,506
Super7800 said:
9211-8i card
Click to expand...
I have never even heard of "INSPUR"

the ART of SERVER is good though. shame you didn't buy your controllers from him. they would have been known good, tested, and updated...
 
Davvo

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
Super7800 said:
I don't think the two other cards are fake, but im not saying they aren't either. They come from a local company that I've done buisness with in the past. They aren't very knowledgeable tho so they could have gotten fake cards and not known it. They advertise everything like technobable keywords Overload. I will take them out tonight and take detailed pictures, perhaps that will shed some light on their authenticity. I'd hate to just keep buying cards when the issue is something else.
Click to expand...
Take a look at this, it might help.

Fake server cards

Some time ago, forum frequenter @artlessknave kindly sent me a pair of LSI HBA cards that had failed to work out in a FreeNAS build. Having suspected that these were fakes, I asked for them to be shipped here for comparison. However, COVID...
www.truenas.com www.truenas.com
 
S

Super7800

Dabbler
Joined
Feb 2, 2021
Messages
16
artlessknave said:
I have never even heard of "INSPUR"

the ART of SERVER is good though. shame you didn't buy your controllers from him. they would have been known good, tested, and updated...
Click to expand...
Ah. So that one's probably a counterfit then. Money thrown away I guess lol. The one I currently have installed seems legit though.

Thanks for the link. I'll use that to compare against when I take em out for inspection.
 
artlessknave

artlessknave

Wizard
Joined
Oct 29, 2016
Messages
1,506
the art of server also has a video about fakes specifically. their videos are pretty much all good.
 
artlessknave

artlessknave

Wizard
Joined
Oct 29, 2016
Messages
1,506
also, that video earlier was only about the boot devices. he does mention "Failover" but that's not done by the HBA, that's done, in the case of trueNAS, by setting up a boot mirror. the LSI configurator part is merely setting which drives the LSI card considers to be boot drives, which controls which drives the LSI card presents to the BIOS as boot drives.
 
HoneyBadger

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
Super7800 said:
i have absolutely no idea what im saying here. I do not know how failover works on truenas yet. a youtube video showing how to set it (dell h200 lsi 9210-8i) up said that. The "lsi configurator" (NOT RAID) allowed me to select the other three drives as "failover drives" incase it failed to boot off of the "primary" boot drive.

here is the ebay link to the cards "sff-8088". they are not flashed to raid since half the disks are functioning correctly (at least i assume so?). sff-8088.
here is the link for the other card that controls the 4 boot disks: dell H200

during setup (install), truenas sees all the disks. the "lsi manager" also sees both disk arrays and all disks. I think that means they are working correctly? Below is what happens when i run the "dmesg" command. Maybe thats helpful?

Code:
Last login: Sun Jan  8 17:55:05 on pts/0
FreeBSD 13.1-RELEASE-p2 n245412-484f039b1d0 TRUENAS

        TrueNAS (c) 2009-2022, iXsystems, Inc.
        All rights reserved.
        TrueNAS code is released under the modified BSD license with some
        files copyrighted by (c) iXsystems, Inc.

        For more information, documentation, help or support, go here:
        http://truenas.com
Welcome to TrueNAS

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@truenas[~]# dmesg
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.1-RELEASE-p2 n245412-484f039b1d0 TRUENAS amd64
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU           E5504  @ 2.00GHz (1999.50-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a  Stepping=5
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID
  TSC: P-state invariant, performance statistics
real memory  = 77309411328 (73728 MB)
avail memory = 75199807488 (71716 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <HP     ProLiant>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
Firmware Warning (ACPI): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20201113/tbfadt-850)
Firmware Warning (ACPI): Invalid length for FADT/Pm2ControlBlock: 32, using default 8 (20201113/tbfadt-850)
ioapic1 <Version 2.0> irqs 24-47
ioapic0 <Version 2.0> irqs 0-23
Launching APs: 3 1 2
random: entropy device external interface
kbd1 at kbdmux0
vtvga0: <VT VGA driver>
aesni0: No AES or SHA support.
padlock0: No ACE support.
acpi0: <HP ProLiant>
acpi0: Power Button (fixed)
acpi0: _OSC failed: AE_BUFFER_OVERFLOW
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
apei0: <ACPI Platform Error Interface> on acpi0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci5: <ACPI PCI bus> on pcib5
mps0: <Avago Technologies (LSI) SAS2008> port 0x6000-0x60ff mem 0xfbff0000-0xfbff3fff,0xfbf80000-0xfbfbffff irq 26 at device 0.0 on pci5
mps0: Firmware: 13.00.57.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib6: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci7: <ACPI PCI bus> on pcib7
mps1: <Avago Technologies (LSI) SAS2008> port 0x5000-0x50ff mem 0xfbef0000-0xfbefffff,0xfbe80000-0xfbebffff irq 30 at device 0.0 on pci7
mps1: Firmware: 07.15.08.00, Driver: 21.02.00.00-fbsd
mps1: IOCCapabilities: 185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR>
pcib8: <ACPI PCI-PCI bridge> at device 8.0 on pci0
pci8: <ACPI PCI bus> on pcib8
ix0: <Intel(R) X520 82599ES (SFI/SFP+)> port 0x4000-0x401f mem 0xfbd80000-0xfbdfffff,0xfbd70000-0xfbd73fff irq 31 at device 0.0 on pci8
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 4 RX queues 4 TX queues
ix0: Using MSI-X interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: 90:e2:ba:30:6c:78
ix0: PCI Express Bus: Speed 5.0GT/s Width x4
ix0: eTrack 0x0001546c PHY FW V65535
ix1: <Intel(R) X520 82599ES (SFI/SFP+)> port 0x4020-0x403f mem 0xfbc80000-0xfbcfffff,0xfbc70000-0xfbc73fff irq 39 at device 0.1 on pci8
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 4 RX queues 4 TX queues
ix1: Using MSI-X interrupts with 5 vectors
ix1: allocated for 4 queues
ix1: allocated for 4 rx queues
ix1: Ethernet address: 90:e2:ba:30:6c:79
ix1: PCI Express Bus: Speed 5.0GT/s Width x4
ix1: eTrack 0x0001546c PHY FW V65535
pcib9: <ACPI PCI-PCI bridge> at device 9.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 10.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached)
pcib11: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci11: <ACPI PCI bus> on pcib11
bce0: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci11
miibus0: <MII bus> on bce0
brgphy0: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce0: Using defaults for TSO: 65518/35/2048
bce0: Ethernet address: 80:c1:6e:23:a2:46
bce0: ASIC (0x57092003);
bce0: link state changed to DOWN
Rev (C0); Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12)
Coal (RX:6,6,18,18; TX:20,20,80,80)
bce1: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf2000000-0xf3ffffff irq 17 at device 0.1 on pci11
miibus1: <MII bus> on bce1
brgphy1: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce1: Using defaults for TSO: 65518/35/2048
bce1: Ethernet address: 80:c1:6e:23:a2:48
bce1: ASIC (0x57092003);
bce1: link state changed to DOWN
Rev (C0); Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib12: <ACPI PCI-PCI bridge> at device 28.2 on pci0
pci12: <ACPI PCI bus> on pcib12
bce2: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf8000000-0xf9ffffff irq 18 at device 0.0 on pci12
miibus2: <MII bus> on bce2
brgphy2: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus2
brgphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce2: Using defaults for TSO: 65518/35/2048
bce2: Ethernet address: 80:c1:6e:23:a2:4a
bce2: ASIC (0x57092003); Rev (C0); Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI)
Coal (RX:6,6,18,18; TX:20,20,80,80)
bce3: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf6000000-0xf7ffffff irq 19 at device 0.1 on pci12
miibus3: <MII bus> on bce3
brgphy3: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus3
brgphy3:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce3: Using defaults for TSO: 65518/35/2048
bce3: Ethernet address: 80:c1:6e:23:a2:4c
bce3: ASIC (0x57092003); Rev (C0); Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib13: <ACPI PCI-PCI bridge> at device 28.4 on pci0
pci13: <ACPI PCI bus> on pcib13
uhci0: <HP iLO Standard Virtual USB controller> port 0x3c00-0x3c1f irq 17 at device 0.4 on pci13
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1: <Intel 82801JI (ICH10) USB controller USB-A> port 0x1000-0x101f irq 20 at device 29.0 on pci0
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2: <Intel 82801JI (ICH10) USB controller USB-B> port 0x1020-0x103f irq 23 at device 29.1 on pci0
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
uhci3: <Intel 82801JI (ICH10) USB controller USB-C> port 0x1040-0x105f irq 22 at device 29.2 on pci0
usbus3 on uhci3
usbus3: 12Mbps Full Speed USB v1.0
uhci4: <Intel 82801JI (ICH10) USB controller USB-F> port 0x1060-0x107f irq 23 at device 29.3 on pci0
usbus4 on uhci4
usbus4: 12Mbps Full Speed USB v1.0
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0xf1bf0000-0xf1bf03ff irq 20 at device 29.7 on pci0
usbus5: EHCI version 1.0
usbus5 on ehci0
usbus5: 480Mbps High Speed USB v2.0
pcib14: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci14: <ACPI PCI bus> on pcib14
vgapci0: <VGA-compatible display> port 0x2000-0x20ff mem 0xe8000000-0xefffffff,0xf1cf0000-0xf1cfffff irq 23 at device 3.0 on pci14
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_tz0: <Thermal Zone> on acpi0
ipmi0: <IPMI System Interface> port 0xca2-0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0.
psm0: model IntelliMouse Explorer, device ID 4
acpi_syscontainer0: <System Container> port 0x2e-0x2f,0x620-0x65f,0x680-0x69f,0x600-0x61f,0x660-0x67f,0x300-0x31f on acpi0
uart0: <Non-standard ns8250 class UART with FIFOs> port 0x3f8-0x3ff irq 4 flags0x10 on acpi0
ichwd0: <Intel ICH10 watchdog timer> on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
ichwd0: <Intel ICH10 watchdog timer> at port 0x930-0x937,0x960-0x97f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff pnpid ORM0000 on isa0
uart1: <Non-standard ns8250 class UART with FIFOs> at port 0x2f8 irq 3 on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr f
device_attach: est0 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr f
device_attach: est1 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr f
device_attach: est2 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr f
device_attach: est3 attach returned 6
Timecounter "TSC" frequency 1999431679 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
ugen0.1: <HP UHCI root HUB> at usbus0
ugen2.1: <Intel UHCI root HUB> at usbus2
ugen3.1: <Intel UHCI root HUB> at usbus3
ugen1.1: <Intel UHCI root HUB> at usbus1
uhub0 on usbus0
uhub0: <HP UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub1 on usbus3
uhub2 on usbus2
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen4.1: <Intel UHCI root HUB> at usbus4
ugen5.1: <Intel EHCI root HUB> at usbus5
uhub3 on usbus1
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
uhub4 on usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
uhub5 on usbus5
uhub5: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
ipmi0: IPMI device rev. 1, firmware rev. 1.89, version 2.0, device support mask0x8f
ipmi0: Number of channels 1
ipmi0: Attached watchdog
ipmi0: Establishing power cycle handler
Trying to mount root from zfs:boot-pool/ROOT/default []...
uhub1: 2 ports with 2 removable, self powered
uhub0: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
ugen0.2: <HP Virtual Keyboard> at usbus0
ukbd0 on uhub0
ukbd0: <Virtual Keyboard > on usbus0
kbd2 at ukbd0
ums0 on uhub0
ums0: <Virtual Mouse> on usbus0
Root mount waiting for: CAM usbus5
Root mount waiting for: CAM usbus5
uhub5: 8 ports with 8 removable, self powered
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
da0 at mps0 bus 0 scbus0 target 8 lun 0
da0: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number YVJSDU3D
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 2861588MB (5860533168 512 byte sectors)
da1 at mps0 bus 0 scbus0 target 9 lun 0
da1: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number YVKMX11K
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 2861588MB (5860533168 512 byte sectors)
da2 at mps0 bus 0 scbus0 target 10 lun 0
da2: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number YVKP6T9K
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 2861588MB (5860533168 512 byte sectors)
da3 at mps0 bus 0 scbus0 target 11 lun 0
da3: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number YVKUA2WK
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 2861588MB (5860533168 512 byte sectors)
da4 at mps0 bus 0 scbus0 target 12 lun 0
da4: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number YVKNEURK
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 2861588MB (5860533168 512 byte sectors)
da7 at mps0 bus 0 scbus0 target 15 lun 0
da7: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number YVKTYMZK
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 2861588MB (5860533168 512 byte sectors)
da9 at mps0 bus 0 scbus0 target 17 lun 0
da9: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da9: Serial Number YVJT1HBK
da9: 600.000MB/s transfers
da9: Command Queueing enabled
da9: 2861588MB (5860533168 512 byte sectors)
da11 at mps0 bus 0 scbus0 target 19 lun 0
da11: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da11: Serial Number YVKP5EAK
da11: 600.000MB/s transfers
da11: Command Queueing enabled
da11: 2861588MB (5860533168 512 byte sectors)
da10 at mps0 bus 0 scbus0 target 18 lun 0
da10: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da10: Serial Number YVKVU0KK
da10: 600.000MB/s transfers
da10: Command Queueing enabled
da10: 2861588MB (5860533168 512 byte sectors)
da13 at mps0 bus 0 scbus0 target 21 lun 0
da13: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da13: Serial Number YVJJ3LMD
da13: 600.000MB/s transfers
da13: Command Queueing enabled
da13: 2861588MB (5860533168 512 byte sectors)
da14 at mps0 bus 0 scbus0 target 22 lun 0
da14: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da14: Serial Number YVKP6UGK
da14: 600.000MB/s transfers
da14: Command Queueing enabled
da14: 2861588MB (5860533168 512 byte sectors)
da12 at mps0 bus 0 scbus0 target 20 lun 0
da12: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da12: Serial Number P9HMRN0W
da12: 600.000MB/s transfers
da12: Command Queueing enabled
da12: 2861588MB (5860533168 512 byte sectors)
da15 at mps0 bus 0 scbus0 target 23 lun 0
da15: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da15: Serial Number YVKTY7GK
da15: 600.000MB/s transfers
da15: Command Queueing enabled
da15: 2861588MB (5860533168 512 byte sectors)
da18 at mps0 bus 0 scbus0 target 47 lun 0
da18: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da18: Serial Number YVKP6T9K
da18: 600.000MB/s transfers
da18: Command Queueing enabled
da18: 2861588MB (5860533168 512 byte sectors)
da17 at mps0 bus 0 scbus0 target 46 lun 0
da17: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da17: Serial Number YVKMX11K
da17: 600.000MB/s transfers
da17: Command Queueing enabled
da17: 2861588MB (5860533168 512 byte sectors)
da19 at mps0 bus 0 scbus0 target 48 lun 0
da19: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da19: Serial Number YVKUA2WK
da19: 600.000MB/s transfers
da19: Command Queueing enabled
da19: 2861588MB (5860533168 512 byte sectors)
da16 at mps0 bus 0 scbus0 target 45 lun 0
da16: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da16: Serial Number YVJSDU3D
da16: 600.000MB/s transfers
da16: Command Queueing enabled
da16: 2861588MB (5860533168 512 byte sectors)
da20 at mps0 bus 0 scbus0 target 49 lun 0
da20: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da20: Serial Number YVKNEURK
da20: 600.000MB/s transfers
da20: Command Queueing enabled
da20: 2861588MB (5860533168 512 byte sectors)
da6 at mps0 bus 0 scbus0 target 14 lun 0
da6: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number P9HN7B4W
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 2861588MB (5860533168 512 byte sectors)
da5 at mps0 bus 0 scbus0 target 13 lun 0
da5: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number P9G7PRRW
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 2861588MB (5860533168 512 byte sectors)
da8 at mps0 bus 0 scbus0 target 16 lun 0
da8: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da8: Serial Number P9J262LW
da8: 600.000MB/s transfers
da8: Command Queueing enabled
da8: 2861588MB (5860533168 512 byte sectors)
ses0 at mps0 bus 0 scbus0 target 44 lun 0
ses0: <Promise J630s 060=> Fixed Enclosure Services SPC-4 SCSI device
ses0: Serial Number TB0038052886
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SES Device
da23 at mps0 bus 0 scbus0 target 52 lun 0
da23: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da23: Serial Number YVKTYMZK
da23: 600.000MB/s transfers
da23: Command Queueing enabled
da23: 2861588MB (5860533168 512 byte sectors)
da25 at mps0 bus 0 scbus0 target 54 lun 0
da25: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da25: Serial Number YVJT1HBK
da25: 600.000MB/s transfers
da25: Command Queueing enabled
da25: 2861588MB (5860533168 512 byte sectors)
ses1 at mps0 bus 0 scbus0 target 81 lun 0
ses1: <Promise J630s 060=> Fixed Enclosure Services SPC-4 SCSI device
ses1: Serial Number TB0038052886
ses1: 600.000MB/s transfers
ses1: Command Queueing enabled
ses1: SES Device
da22 at mps0 bus 0 scbus0 target 51 lun 0
da22: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da22: Serial Number P9HN7B4W
da22: 600.000MB/s transfers
da22: Command Queueing enabled
da22: 2861588MB (5860533168 512 byte sectors)
da21 at mps0 bus 0 scbus0 target 50 lun 0
da21: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da21: Serial Number P9G7PRRW
da21: 600.000MB/s transfers
da21: Command Queueing enabled
da21: 2861588MB (5860533168 512 byte sectors)
da27 at mps0 bus 0 scbus0 target 56 lun 0
da27: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da27: Serial Number YVKP5EAK
da27: 600.000MB/s transfers
da27: Command Queueing enabled
da27: 2861588MB (5860533168 512 byte sectors)
da26 at mps0 bus 0 scbus0 target 55 lun 0
da26: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da26: Serial Number YVKVU0KK
da26: 600.000MB/s transfers
da26: Command Queueing enabled
da26: 2861588MB (5860533168 512 byte sectors)
da24 at mps0 bus 0 scbus0 target 53 lun 0
da24: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da24: Serial Number P9J262LW
da24: 600.000MB/s transfers
da24: Command Queueing enabled
da24: 2861588MB (5860533168 512 byte sectors)
da31 at mps0 bus 0 scbus0 target 60 lun 0
da31: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da31: Serial Number YVKTY7GK
da31: 600.000MB/s transfers
da31: Command Queueing enabled
da31: 2861588MB (5860533168 512 byte sectors)
da29 at mps0 bus 0 scbus0 target 58 lun 0
da29: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da29: Serial Number YVJJ3LMD
da29: 600.000MB/s transfers
da29: Command Queueing enabled
da29: 2861588MB (5860533168 512 byte sectors)
da30 at mps0 bus 0 scbus0 target 59 lun 0
da30: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da30: Serial Number YVKP6UGK
da30: 600.000MB/s transfers
da30: Command Queueing enabled
da30: 2861588MB (5860533168 512 byte sectors)
da28 at mps0 bus 0 scbus0 target 57 lun 0
da28: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da28: Serial Number P9HMRN0W
da28: 600.000MB/s transfers
da28: Command Queueing enabled
da28: 2861588MB (5860533168 512 byte sectors)
da34 at mps1 bus 0 scbus1 target 9 lun 0
da34: <HP DG0146FAMWL HPDC> Fixed Direct Access SPC-3 SCSI device
da34: Serial Number 3SD13EP400009005J5LK
da34: 600.000MB/s transfers
da34: Command Queueing enabled
da34: 140014MB (286749488 512 byte sectors)
da35 at mps1 bus 0 scbus1 target 10 lun 0
da35: <HP DG0146FAMWL HPDB> Fixed Direct Access SPC-3 SCSI device
da35: Serial Number 3SD0QESR00009953RW2U
da35: 600.000MB/s transfers
da35: Command Queueing enabled
da35: 140014MB (286749488 512 byte sectors)
da32 at mps1 bus 0 scbus1 target 7 lun 0
da32: <HP DG146ABAB4 HPDC> Fixed Direct Access SPC-3 SCSI device
da32: Serial Number 3NM4BY200000982745Y8
da32: 300.000MB/s transfers
da32: Command Queueing enabled
da32: 140014MB (286749488 512 byte sectors)
da33 at mps1 bus 0 scbus1 target 8 lun 0
da33: <HP DG146BB976 HPDC> Fixed Direct Access SPC-3 SCSI device
da33: Serial Number 3NM5606F000098388V18
da33: 300.000MB/s transfers
da33: Command Queueing enabled
da33: 140014MB (286749488 512 byte sectors)
ses0: da0,pass0 in Array Device Slot 1, SAS Slot: 1+ phys at slot 1
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03e9b4e86
ses0: da1,pass1 in Array Device Slot 2, SAS Slot: 1+ phys at slot 2
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecd4b32
ses1: da16,pass17 in Array Device Slot 1, SAS Slot: 1+ phys at slot 1
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03e9b4e85
ses0: da2,pass2 in Array Device Slot 3, SAS Slot: 1+ phys at slot 3
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecfaf16
ses1: da17,pass18 in Array Device Slot 2, SAS Slot: 1+ phys at slot 2
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecd4b31
ses0: da3,pass3 in Array Device Slot 4, SAS Slot: 1+ phys at slot 4
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed726b6
ses1: da18,pass19 in Array Device Slot 3, SAS Slot: 1+ phys at slot 3
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecfaf15
ses0: da4,pass4 in Array Device Slot 5, SAS Slot: 1+ phys at slot 5
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ece4766
ses1: da19,pass20 in Array Device Slot 4, SAS Slot: 1+ phys at slot 4
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed726b5
ses0: da5,pass5 in Array Device Slot 6, SAS Slot: 1+ phys at slot 6
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca0580e00ba
ses1: da20,pass21 in Array Device Slot 5, SAS Slot: 1+ phys at slot 5
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ece4765
ses0: da6,pass6 in Array Device Slot 7, SAS Slot: 1+ phys at slot 7
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca0585d2a36
ses1: da21,pass22 in Array Device Slot 6, SAS Slot: 1+ phys at slot 6
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca0580e00b9
ses0: da7,pass7 in Array Device Slot 8, SAS Slot: 1+ phys at slot 8
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed67ace
ses1: da22,pass23 in Array Device Slot 7, SAS Slot: 1+ phys at slot 7
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca0585d2a35
ses0: da8,pass8 in Array Device Slot 9, SAS Slot: 1+ phys at slot 9
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca05874ba5a
ses1: da23,pass24 in Array Device Slot 8, SAS Slot: 1+ phys at slot 8
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed67acd
ses0: da9,pass9 in Array Device Slot 10, SAS Slot: 1+ phys at slot 10
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03e9c761e
ses1: da24,pass25 in Array Device Slot 9, SAS Slot: 1+ phys at slot 9
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca05874ba59
ses0: da10,pass10 in Array Device Slot 11, SAS Slot: 1+ phys at slot 11
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed9d84e
ses1: da25,pass26 in Array Device Slot 10, SAS Slot: 1+ phys at slot 10
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03e9c761d
ses0: da11,pass11 in Array Device Slot 12, SAS Slot: 1+ phys at slot 12
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecf9ac2
ses1: da26,pass27 in Array Device Slot 11, SAS Slot: 1+ phys at slot 11
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed9d84d
ses0: da12,pass12 in Array Device Slot 13, SAS Slot: 1+ phys at slot 13
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca0585c3e42
ses1: da27,pass28 in Array Device Slot 12, SAS Slot: 1+ phys at slot 12
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecf9ac1
ses0: da13,pass13 in Array Device Slot 14, SAS Slot: 1+ phys at slot 14
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03e8e09da
ses1: da28,pass29 in Array Device Slot 13, SAS Slot: 1+ phys at slot 13
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca0585c3e41
ses0: da14,pass14 in Array Device Slot 15, SAS Slot: 1+ phys at slot 15
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecfafa6
ses1: da29,pass30 in Array Device Slot 14, SAS Slot: 1+ phys at slot 14
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03e8e09d9
ses0: da15,pass15 in Array Device Slot 16, SAS Slot: 1+ phys at slot 16
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed674be
ses1: da30,pass31 in Array Device Slot 15, SAS Slot: 1+ phys at slot 15
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecfafa5
ses1: da31,pass32 in Array Device Slot 16, SAS Slot: 1+ phys at slot 16
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed674bd
lo0: link state changed to UP
GEOM_MULTIPATH: disk13 created
GEOM_MULTIPATH: da28 added to disk13
GEOM_MULTIPATH: da28 is now active path in disk13
GEOM_MULTIPATH: disk15 created
GEOM_MULTIPATH: da30 added to disk15
GEOM_MULTIPATH: da30 is now active path in disk15
GEOM_MULTIPATH: disk14 created
GEOM_MULTIPATH: da29 added to disk14
GEOM_MULTIPATH: da29 is now active path in disk14
GEOM_MULTIPATH: disk16 created
GEOM_MULTIPATH: da31 added to disk16
GEOM_MULTIPATH: da31 is now active path in disk16
GEOM_MULTIPATH: disk9 created
GEOM_MULTIPATH: da24 added to disk9
GEOM_MULTIPATH: da24 is now active path in disk9
GEOM_MULTIPATH: disk11 created
GEOM_MULTIPATH: da26 added to disk11
GEOM_MULTIPATH: da26 is now active path in disk11
GEOM_MULTIPATH: disk6 created
GEOM_MULTIPATH: da21 added to disk6
GEOM_MULTIPATH: da21 is now active path in disk6
GEOM_MULTIPATH: disk12 created
GEOM_MULTIPATH: da27 added to disk12
GEOM_MULTIPATH: da27 is now active path in disk12
GEOM_MULTIPATH: disk7 created
GEOM_MULTIPATH: da22 added to disk7
GEOM_MULTIPATH: da22 is now active path in disk7
GEOM_MULTIPATH: disk10 created
GEOM_MULTIPATH: da25 added to disk10
GEOM_MULTIPATH: da25 is now active path in disk10
GEOM_MULTIPATH: disk8 created
GEOM_MULTIPATH: da23 added to disk8
GEOM_MULTIPATH: da23 is now active path in disk8
GEOM_MULTIPATH: da8 added to disk9
GEOM_MULTIPATH: da5 added to disk6
GEOM_MULTIPATH: da6 added to disk7
GEOM_MULTIPATH: disk5 created
GEOM_MULTIPATH: da20 added to disk5
GEOM_MULTIPATH: da20 is now active path in disk5
GEOM_MULTIPATH: disk1 created
GEOM_MULTIPATH: da16 added to disk1
GEOM_MULTIPATH: da16 is now active path in disk1
GEOM_MULTIPATH: disk4 created
GEOM_MULTIPATH: da19 added to disk4
GEOM_MULTIPATH: da19 is now active path in disk4
GEOM_MULTIPATH: disk2 created
GEOM_MULTIPATH: da17 added to disk2
GEOM_MULTIPATH: da17 is now active path in disk2
GEOM_MULTIPATH: disk3 created
GEOM_MULTIPATH: da18 added to disk3
GEOM_MULTIPATH: da18 is now active path in disk3
GEOM_MULTIPATH: da15 added to disk16
GEOM_MULTIPATH: da12 added to disk13
GEOM_MULTIPATH: da14 added to disk15
GEOM_MULTIPATH: da13 added to disk14
GEOM_MULTIPATH: da10 added to disk11
GEOM_MULTIPATH: da11 added to disk12
GEOM_MULTIPATH: da9 added to disk10
GEOM_MULTIPATH: da7 added to disk8
GEOM_MULTIPATH: da4 added to disk5
GEOM_MULTIPATH: da3 added to disk4
GEOM_MULTIPATH: da2 added to disk3
GEOM_MULTIPATH: da1 added to disk2
GEOM_MULTIPATH: da0 added to disk1
CPU: Intel(R) Xeon(R) CPU           E5504  @ 2.00GHz (1999.43-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a  Stepping=5
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features3=0x9c000000<IBPB,STIBP,L1DFL,SSBD>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID
  TSC: P-state invariant, performance statistics
bce2: link state changed to DOWN
bce3: link st
Click to expand...

Hello!

I've taken a moment to dig through your dmesg output here - we've got a few things in play that I'd like to call attention to. Bear with me here, and apologies in advance for the length and potential technical depth here.

Your HGST HDDs are SAS devices - this means they have two available data paths. Disks da0 through da15 are the same devices as da16 through da31 - they're getting combined by the SAS multipath driver into a single disk. See the example of the da0 and da16 devices below being identified with the same serial number, and combined into the same disk:

Code:
da0 at mps0 bus 0 scbus0 target 8 lun 0
da0: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number YVJSDU3D
da16 at mps0 bus 0 scbus0 target 45 lun 0
da16: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da16: Serial Number YVJSDU3D
GEOM_MULTIPATH: disk1 created
GEOM_MULTIPATH: da16 added to disk1
GEOM_MULTIPATH: da16 is now active path in disk1
GEOM_MULTIPATH: da0 added to disk1


So, it's actually only seeing chassis of drives, and it's actually picking up the "secondary" path first. Assuming you have the same SAS drives in both of the JBODs, I'd expect to see another set of 16x2=32 daX devices.

I'm betting the reason we only see one is related to the PCIe-slot-to-CPU-socket relationship.

Each CPU has a limited number of PCIe lanes, and given the configuration for your DL380 G7 (three HBAs and a 10Gbps NIC) I imagine you've inserted your second external HBA into a slot that's driven from the second CPU socket. You may have to look at purchasing a second CPU (although, I would consider updating your BIOS/firmware and using 5600-series Xeon chips for better performance, both in an absolute speed and a performance-per-watt standpoint.)

Speaking of firmware:
Code:
mps0: Firmware: 13.00.57.00, Driver: 21.02.00.00-fbsd
mps1: Firmware: 07.15.08.00, Driver: 21.02.00.00-fbsd

TrueNAS will likely alert you to this, but I would recommend updating your HBA firmware as well. Please check that you're loading appropriate (8i vs 8e) firmware onto the respective cards - have only one plugged in at a time if you want to be extra safe.

Super7800 said:
I figured the 10K SAS drives [in the boot pool] would have the speed, but perhaps not.
Click to expand...

Spinning drives for a boot-pool should be sufficient here. SSDs will obviously be faster when performing system updates, but are not strictly necessary. The failover is related to redundant/failover boot devices in the LSI BIOS as discussed by @artlessknave

JBOD expansion/cabling
We're getting a little technically deep now, but I believe that you currently have each JBOD connected to a single HBA with two cables? This could be changed to a "chained" method where each HBA in your server will have a path to each JBOD shelf, and then back to the other HBA - this is more complex, but allows for the failure of a single HBA without causing an entire JBOD shelf to go offline.
 
S

Super7800

Dabbler
Joined
Feb 2, 2021
Messages
16
HoneyBadger said:
I'm betting the reason we only see one is related to the PCIe-slot-to-CPU-socket relationship.

Each CPU has a limited number of PCIe lanes, and given the configuration for your DL380 G7 (three HBAs and a 10Gbps NIC) I imagine you've inserted your second external HBA into a slot that's driven from the second CPU socket. You may have to look at purchasing a second CPU (although, I would consider updating your BIOS/firmware and using 5600-series Xeon chips for better performance, both in an absolute speed and a performance-per-watt standpoint.)
Click to expand...

Ok. I have just ordered a pair of x5687 cpus (4 core 3.6ghz). These are what I run in all my other hp g7s so figured I'd keep it the same here too. I'll probably install another set of ram aswell (double the current amount).

The jbods allow Daisy chaining (i.e. I could actually get away with one raid card I believe), but I think each one with a dedicated card and two cables is faster?

You are correct. One card per jbod two cables each. Interesting with the chaining. So your saying each hba should have one cable going to each jbod? Would it still see all the disks? If so that sounds like a great idea to me.

Thanks you for the help!
 
artlessknave

artlessknave

Wizard
Joined
Oct 29, 2016
Messages
1,506
Super7800 said:
but I think each one with a dedicated card and two cables is faster?
Click to expand...
technically yes but you are running spinners. you can run the whole thing on 1 sas1 card and probably still get full speeds...
 
S

Super7800

Dabbler
Joined
Feb 2, 2021
Messages
16
Installed the two new CPUs, and the disks still arent showing up. However, the numbering is all off now. And come to think of it, how could it have been a cpu issue? all the disks showed up fine during install and in the LSI configurator. Its only once installed that they are missing.

Here is DMESG:
Code:
Last login: Fri Jan 13 17:35:03 on pts/0
FreeBSD 13.1-RELEASE-p2 n245412-484f039b1d0 TRUENAS

        TrueNAS (c) 2009-2022, iXsystems, Inc.
        All rights reserved.
        TrueNAS code is released under the modified BSD license with some
        files copyrighted by (c) iXsystems, Inc.

        For more information, documentation, help or support, go here:
        http://truenas.com
Welcome to TrueNAS

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@truenas[~]# dmesg
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.1-RELEASE-p2 n245412-484f039b1d0 TRUENAS amd64
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU           X5687  @ 3.60GHz (3599.06-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 94489280512 (90112 MB)
avail memory = 91937742848 (87678 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <HP     ProLiant>
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 hardware threads
random: unblocking device.
Firmware Warning (ACPI): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20201113/tbfadt-850)
Firmware Warning (ACPI): Invalid length for FADT/Pm2ControlBlock: 32, using default 8 (20201113/tbfadt-850)
ioapic1 <Version 2.0> irqs 24-47
ioapic0 <Version 2.0> irqs 0-23
Launching APs: 1 3 2 5 6 4 14 13 8 9 10 7 11 15 12
random: entropy device external interface
kbd1 at kbdmux0
vtvga0: <VT VGA driver>
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
padlock0: No ACE support.
acpi0: <HP ProLiant>
acpi0: Power Button (fixed)
acpi0: _OSC failed: AE_BUFFER_OVERFLOW
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
apei0: <ACPI Platform Error Interface> on acpi0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci5: <ACPI PCI bus> on pcib5
mps0: <Avago Technologies (LSI) SAS2008> port 0x6000-0x60ff mem 0xfbff0000-0xfbff3fff,0xfbf80000-0xfbfbffff irq 26 at device 0.0 on pci5
mps0: Firmware: 13.00.57.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib6: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci7: <ACPI PCI bus> on pcib7
mps1: <Avago Technologies (LSI) SAS2008> port 0x5000-0x50ff mem 0xfbef0000-0xfbefffff,0xfbe80000-0xfbebffff irq 30 at device 0.0 on pci7
mps1: Firmware: 07.15.08.00, Driver: 21.02.00.00-fbsd
mps1: IOCCapabilities: 185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR>
pcib8: <ACPI PCI-PCI bridge> at device 8.0 on pci0
pci8: <ACPI PCI bus> on pcib8
ix0: <Intel(R) X520 82599ES (SFI/SFP+)> port 0x4000-0x401f mem 0xfbd80000-0xfbdfffff,0xfbd70000-0xfbd73fff irq 31 at device 0.0 on pci8
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 8 RX queues 8 TX queues
ix0: Using MSI-X interrupts with 9 vectors
ix0: allocated for 8 queues
ix0: allocated for 8 rx queues
ix0: Ethernet address: 90:e2:ba:30:6c:78
ix0: PCI Express Bus: Speed 5.0GT/s Width x4
ix0: eTrack 0x0001546c PHY FW V65535
ix1: <Intel(R) X520 82599ES (SFI/SFP+)> port 0x4020-0x403f mem 0xfbc80000-0xfbcfffff,0xfbc70000-0xfbc73fff irq 39 at device 0.1 on pci8
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 8 RX queues 8 TX queues
ix1: Using MSI-X interrupts with 9 vectors
ix1: allocated for 8 queues
ix1: allocated for 8 rx queues
ix1: Ethernet address: 90:e2:ba:30:6c:79
ix1: PCI Express Bus: Speed 5.0GT/s Width x4
ix1: eTrack 0x0001546c PHY FW V65535
pcib9: <ACPI PCI-PCI bridge> at device 9.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 10.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached)
pcib11: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci11: <ACPI PCI bus> on pcib11
bce0: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci11
miibus0: <MII bus> on bce0
brgphy0: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce0: Using defaults for TSO: 65518/35/2048
bce0: Ethernet address: 80:c1:6e:23:a2:46
bce0: ASIC (0x57092003); Rev (C0);
bce0: link state changed to DOWN
Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12)
Coal (RX:6,6,18,18; TX:20,20,80,80)
bce1: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf2000000-0xf3ffffff irq 17 at device 0.1 on pci11
miibus1: <MII bus> on bce1
brgphy1: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce1: Using defaults for TSO: 65518/35/2048
bce1: Ethernet address: 80:c1:6e:23:a2:48
bce1: ASIC (0x57092003);
bce1: link state changed to DOWN
Rev (C0); Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib12: <ACPI PCI-PCI bridge> at device 28.2 on pci0
pci12: <ACPI PCI bus> on pcib12
bce2: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf8000000-0xf9ffffff irq 18 at device 0.0 on pci12
miibus2: <MII bus> on bce2
brgphy2: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus2
brgphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce2: Using defaults for TSO: 65518/35/2048
bce2: Ethernet address: 80:c1:6e:23:a2:4a
bce2: ASIC (0x57092003); Rev (C0); Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI)
Coal (RX:6,6,18,18; TX:20,20,80,80)
bce3: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf6000000-0xf7ffffff irq 19 at device 0.1 on pci12
miibus3: <MII bus> on bce3
brgphy3: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus3
brgphy3:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce3: Using defaults for TSO: 65518/35/2048
bce3: Ethernet address: 80:c1:6e:23:a2:4c
bce3: ASIC (0x57092003); Rev (C0); Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib13: <ACPI PCI-PCI bridge> at device 28.4 on pci0
pci13: <ACPI PCI bus> on pcib13
uhci0: <HP iLO Standard Virtual USB controller> port 0x3c00-0x3c1f irq 17 at device 0.4 on pci13
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1: <Intel 82801JI (ICH10) USB controller USB-A> port 0x1000-0x101f irq 20 at device 29.0 on pci0
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2: <Intel 82801JI (ICH10) USB controller USB-B> port 0x1020-0x103f irq 23 at device 29.1 on pci0
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
uhci3: <Intel 82801JI (ICH10) USB controller USB-C> port 0x1040-0x105f irq 22 at device 29.2 on pci0
usbus3 on uhci3
usbus3: 12Mbps Full Speed USB v1.0
uhci4: <Intel 82801JI (ICH10) USB controller USB-F> port 0x1060-0x107f irq 23 at device 29.3 on pci0
usbus4 on uhci4
usbus4: 12Mbps Full Speed USB v1.0
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0xf1bf0000-0xf1bf03ff irq 20 at device 29.7 on pci0
usbus5: EHCI version 1.0
usbus5 on ehci0
usbus5: 480Mbps High Speed USB v2.0
pcib14: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci14: <ACPI PCI bus> on pcib14
vgapci0: <VGA-compatible display> port 0x2000-0x20ff mem 0xe8000000-0xefffffff,0xf1cf0000-0xf1cfffff irq 23 at device 3.0 on pci14
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_tz0: <Thermal Zone> on acpi0
ipmi0: <IPMI System Interface> port 0xca2-0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0.
psm0: model IntelliMouse Explorer, device ID 4
acpi_syscontainer0: <System Container> port 0x2e-0x2f,0x620-0x65f,0x680-0x69f,0x600-0x61f,0x660-0x67f,0x300-0x31f on acpi0
uart0: <Non-standard ns8250 class UART with FIFOs> port 0x3f8-0x3ff irq 4 flags0x10 on acpi0
ichwd0: <Intel ICH10 watchdog timer> on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
ichwd0: <Intel ICH10 watchdog timer> at port 0x930-0x937,0x960-0x97f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff pnpid ORM0000 on isa0
uart1: <Non-standard ns8250 class UART with FIFOs> at port 0x2f8 irq 3 on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est0 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est1 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est2 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est3 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est4 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est5 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est6 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est7 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est8 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est9 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est10 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est11 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est12 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est13 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est14 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est15 attach returned 6
Timecounter "TSC-low" frequency 1799488487 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ugen1.1: <Intel UHCI root HUB> at usbus1
ugen3.1: <Intel UHCI root HUB> at usbus3
ugen2.1: <Intel UHCI root HUB> at usbus2
ugen4.1: <Intel UHCI root HUB> at usbus4
ugen5.1: <Intel EHCI root HUB> at usbus5
ugen0.1: <HP UHCI root HUB> at usbus0
uhub0 on usbus1
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
uhub1 on usbus3
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
uhub2 on usbus4
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
uhub3 on usbus5
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
uhub4 on usbus2
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
uhub5 on usbus0
uhub5: <HP UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ipmi0: IPMI device rev. 1, firmware rev. 1.89, version 2.0, device support mask0x8f
ipmi0: Number of channels 1
ipmi0: Attached watchdog
ipmi0: Establishing power cycle handler
Trying to mount root from zfs:boot-pool/ROOT/default []...
uhub0: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
ugen0.2: <HP Virtual Keyboard> at usbus0
ukbd0 on uhub5
ukbd0: <Virtual Keyboard > on usbus0
kbd2 at ukbd0
ums0 on uhub5
ums0: <Virtual Mouse> on usbus0
Root mount waiting for: CAM usbus5
Root mount waiting for: CAM usbus5
uhub3: 8 ports with 8 removable, self powered
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
da0 at mps0 bus 0 scbus0 target 8 lun 0
da0: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number YVJSDU3D
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 2861588MB (5860533168 512 byte sectors)
da2 at mps0 bus 0 scbus0 target 10 lun 0
da2: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number YVKP6T9K
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 2861588MB (5860533168 512 byte sectors)
da3 at mps0 bus 0 scbus0 target 11 lun 0
da3: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number YVKUA2WK
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 2861588MB (5860533168 512 byte sectors)
da1 at mps0 bus 0 scbus0 target 9 lun 0
da1: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number YVKMX11K
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 2861588MB (5860533168 512 byte sectors)
da11 at mps0 bus 0 scbus0 target 19 lun 0
da11: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da11: Serial Number YVKP5EAK
da11: 600.000MB/s transfers
da11: Command Queueing enabled
da11: 2861588MB (5860533168 512 byte sectors)
da4 at mps0 bus 0 scbus0 target 12 lun 0
da4: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number YVKNEURK
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 2861588MB (5860533168 512 byte sectors)
da9 at mps0 bus 0 scbus0 target 17 lun 0
da9: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da9: Serial Number YVJT1HBK
da9: 600.000MB/s transfers
da9: Command Queueing enabled
da9: 2861588MB (5860533168 512 byte sectors)
da5 at mps0 bus 0 scbus0 target 13 lun 0
da5: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number P9G7PRRW
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 2861588MB (5860533168 512 byte sectors)
da14 at mps0 bus 0 scbus0 target 22 lun 0
da14: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da14: Serial Number YVKP6UGK
da14: 600.000MB/s transfers
da14: Command Queueing enabled
da14: 2861588MB (5860533168 512 byte sectors)
da15 at mps0 bus 0 scbus0 target 23 lun 0
da15: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da15: Serial Number YVKTY7GK
da15: 600.000MB/s transfers
da15: Command Queueing enabled
da15: 2861588MB (5860533168 512 byte sectors)
da8 at mps0 bus 0 scbus0 target 16 lun 0
da8: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da8: Serial Number P9J262LW
da8: 600.000MB/s transfers
da8: Command Queueing enabled
da8: 2861588MB (5860533168 512 byte sectors)
da6 at mps0 bus 0 scbus0 target 14 lun 0
da6: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number P9HN7B4W
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 2861588MB (5860533168 512 byte sectors)
da13 at mps0 bus 0 scbus0 target 21 lun 0
da13: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da13: Serial Number YVJJ3LMD
da13: 600.000MB/s transfers
da13: Command Queueing enabled
da13: 2861588MB (5860533168 512 byte sectors)
da17 at mps0 bus 0 scbus0 target 46 lun 0
da17: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da17: Serial Number YVKMX11K
da17: 600.000MB/s transfers
da17: Command Queueing enabled
da17: 2861588MB (5860533168 512 byte sectors)
da18 at mps0 bus 0 scbus0 target 47 lun 0
da18: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da18: Serial Number YVKP6T9K
da18: 600.000MB/s transfers
da18: Command Queueing enabled
da18: 2861588MB (5860533168 512 byte sectors)
da20 at mps0 bus 0 scbus0 target 49 lun 0
da20: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da20: Serial Number YVKNEURK
da20: 600.000MB/s transfers
da20: Command Queueing enabled
da20: 2861588MB (5860533168 512 byte sectors)
da23 at mps0 bus 0 scbus0 target 52 lun 0
da23: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da23: Serial Number YVKTYMZK
da23: 600.000MB/s transfers
da23: Command Queueing enabled
da23: 2861588MB (5860533168 512 byte sectors)
da27 at mps0 bus 0 scbus0 target 56 lun 0
da27: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da27: Serial Number YVKP5EAK
da27: 600.000MB/s transfers
da27: Command Queueing enabled
da27: 2861588MB (5860533168 512 byte sectors)
da7 at mps0 bus 0 scbus0 target 15 lun 0
da7: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number YVKTYMZK
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 2861588MB (5860533168 512 byte sectors)
da26 at mps0 bus 0 scbus0 target 55 lun 0
da26: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da26: Serial Number YVKVU0KK
da26: 600.000MB/s transfers
da26: Command Queueing enabled
da26: 2861588MB (5860533168 512 byte sectors)
da30 at mps0 bus 0 scbus0 target 59 lun 0
da30: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da30: Serial Number YVKP6UGK
da30: 600.000MB/s transfers
da30: Command Queueing enabled
da30: 2861588MB (5860533168 512 byte sectors)
da29 at mps0 bus 0 scbus0 target 58 lun 0
da29: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da29: Serial Number YVJJ3LMD
da29: 600.000MB/s transfers
da29: Command Queueing enabled
da29: 2861588MB (5860533168 512 byte sectors)
da12 at mps0 bus 0 scbus0 target 20 lun 0
da12: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da12: Serial Number P9HMRN0W
da12: 600.000MB/s transfers
da12: Command Queueing enabled
da12: 2861588MB (5860533168 512 byte sectors)
da19 at mps0 bus 0 scbus0 target 48 lun 0
da19: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da19: Serial Number YVKUA2WK
da19: 600.000MB/s transfers
da19: Command Queueing enabled
da19: 2861588MB (5860533168 512 byte sectors)
da24 at mps0 bus 0 scbus0 target 53 lun 0
da24: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da24: Serial Number P9J262LW
da24: 600.000MB/s transfers
da24: Command Queueing enabled
da24: 2861588MB (5860533168 512 byte sectors)
da16 at mps0 bus 0 scbus0 target 45 lun 0
da16: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da16: Serial Number YVJSDU3D
da16: 600.000MB/s transfers
da16: Command Queueing enabled
da16: 2861588MB (5860533168 512 byte sectors)
da21 at mps0 bus 0 scbus0 target 50 lun 0
da21: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da21: Serial Number P9G7PRRW
da21: 600.000MB/s transfers
da21: Command Queueing enabled
da21: 2861588MB (5860533168 512 byte sectors)
da34 at mps1 bus 0 scbus1 target 9 lun 0
da34: <HP DG0146FAMWL HPDC> Fixed Direct Access SPC-3 SCSI device
da34: Serial Number 3SD13EP400009005J5LK
da34: 600.000MB/s transfers
da34: Command Queueing enabled
da34: 140014MB (286749488 512 byte sectors)
da10 at mps0 bus 0 scbus0 target 18 lun 0
da10: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da10: Serial Number YVKVU0KK
da10: 600.000MB/s transfers
da10: Command Queueing enabled
da10: 2861588MB (5860533168 512 byte sectors)
da25 at mps0 bus 0 scbus0 target 54 lun 0
da25: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da25: Serial Number YVJT1HBK
da25: 600.000MB/s transfers
da25: Command Queueing enabled
da25: 2861588MB (5860533168 512 byte sectors)
ses0 at mps0 bus 0 scbus0 target 44 lun 0
ses0: <Promise J630s 060=> Fixed Enclosure Services SPC-4 SCSI device
ses0: Serial Number TB0038052886
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SES Device
da35 at mps1 bus 0 scbus1 target 10 lun 0
da35: <HP DG0146FAMWL HPDB> Fixed Direct Access SPC-3 SCSI device
da35: Serial Number 3SD0QESR00009953RW2U
da35: 600.000MB/s transfers
da35: Command Queueing enabled
da35: 140014MB (286749488 512 byte sectors)
da31 at mps0 bus 0 scbus0 target 60 lun 0
da31: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da31: Serial Number YVKTY7GK
da31: 600.000MB/s transfers
da31: Command Queueing enabled
da31: 2861588MB (5860533168 512 byte sectors)
da22 at mps0 bus 0 scbus0 target 51 lun 0
da22: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da22: Serial Number P9HN7B4W
da22: 600.000MB/s transfers
da22: Command Queueing enabled
da22: 2861588MB (5860533168 512 byte sectors)
da28 at mps0 bus 0 scbus0 target 57 lun 0
da28: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da28: Serial Number P9HMRN0W
da28: 600.000MB/s transfers
da28: Command Queueing enabled
da28: 2861588MB (5860533168 512 byte sectors)
ses1 at mps0 bus 0 scbus0 target 81 lun 0
ses1: <Promise J630s 060=> Fixed Enclosure Services SPC-4 SCSI device
ses1: Serial Number TB0038052886
ses1: 600.000MB/s transfers
ses1: Command Queueing enabled
ses1: SES Device
da32 at mps1 bus 0 scbus1 target 7 lun 0
da32: <HP DG146ABAB4 HPDC> Fixed Direct Access SPC-3 SCSI device
da32: Serial Number 3NM4BY200000982745Y8
da32: 300.000MB/s transfers
da32: Command Queueing enabled
da32: 140014MB (286749488 512 byte sectors)
da33 at mps1 bus 0 scbus1 target 8 lun 0
da33: <HP DG146BB976 HPDC> Fixed Direct Access SPC-3 SCSI device
da33: Serial Number 3NM5606F000098388V18
da33: 300.000MB/s transfers
da33: Command Queueing enabled
da33: 140014MB (286749488 512 byte sectors)
ses0: da0,pass0 in Array Device Slot 1, SAS Slot: 1+ phys at slot 1
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03e9b4e86
ses0: da1,pass1 in Array Device Slot 2, SAS Slot: 1+ phys at slot 2
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecd4b32
ses0: da2,pass2 in Array Device Slot 3, SAS Slot: 1+ phys at slot 3
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecfaf16
ses0: da3,pass3 in Array Device Slot 4, SAS Slot: 1+ phys at slot 4
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed726b6
ses0: da4,pass4 in Array Device Slot 5, SAS Slot: 1+ phys at slot 5
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ece4766
ses0: da5,pass5 in Array Device Slot 6, SAS Slot: 1+ phys at slot 6
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca0580e00ba
ses1: da16,pass17 in Array Device Slot 1, SAS Slot: 1+ phys at slot 1
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03e9b4e85
ses0: da6,pass6 in Array Device Slot 7, SAS Slot: 1+ phys at slot 7
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca0585d2a36
ses1: da17,pass18 in Array Device Slot 2, SAS Slot: 1+ phys at slot 2
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecd4b31
ses0: da7,pass7 in Array Device Slot 8, SAS Slot: 1+ phys at slot 8
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed67ace
ses1: da18,pass19 in Array Device Slot 3, SAS Slot: 1+ phys at slot 3
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecfaf15
ses0: da8,pass8 in Array Device Slot 9, SAS Slot: 1+ phys at slot 9
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca05874ba5a
ses1: da19,pass20 in Array Device Slot 4, SAS Slot: 1+ phys at slot 4
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed726b5
ses0: da9,pass9 in Array Device Slot 10, SAS Slot: 1+ phys at slot 10
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03e9c761e
ses1: da20,pass21 in Array Device Slot 5, SAS Slot: 1+ phys at slot 5
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ece4765
ses0: da10,pass10 in Array Device Slot 11, SAS Slot: 1+ phys at slot 11
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed9d84e
ses1: da21,pass22 in Array Device Slot 6, SAS Slot: 1+ phys at slot 6
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca0580e00b9
ses0: da11,pass11 in Array Device Slot 12, SAS Slot: 1+ phys at slot 12
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecf9ac2
ses1: da22,pass23 in Array Device Slot 7, SAS Slot: 1+ phys at slot 7
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca0585d2a35
ses0: da12,pass12 in Array Device Slot 13, SAS Slot: 1+ phys at slot 13
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca0585c3e42
ses1: da23,pass24 in Array Device Slot 8, SAS Slot: 1+ phys at slot 8
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed67acd
ses0: da13,pass13 in Array Device Slot 14, SAS Slot: 1+ phys at slot 14
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03e8e09da
ses1: da24,pass25 in Array Device Slot 9, SAS Slot: 1+ phys at slot 9
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca05874ba59
ses0: da14,pass14 in Array Device Slot 15, SAS Slot: 1+ phys at slot 15
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecfafa6
ses1: da25,pass26 in Array Device Slot 10, SAS Slot: 1+ phys at slot 10
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03e9c761d
ses0: da15,pass15 in Array Device Slot 16, SAS Slot: 1+ phys at slot 16
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed674be
ses1: da26,pass27 in Array Device Slot 11, SAS Slot: 1+ phys at slot 11
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed9d84d
ses1: da27,pass28 in Array Device Slot 12, SAS Slot: 1+ phys at slot 12
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecf9ac1
ses1: da28,pass29 in Array Device Slot 13, SAS Slot: 1+ phys at slot 13
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca0585c3e41
ses1: da29,pass30 in Array Device Slot 14, SAS Slot: 1+ phys at slot 14
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03e8e09d9
ses1: da30,pass31 in Array Device Slot 15, SAS Slot: 1+ phys at slot 15
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecfafa5
ses1: da31,pass32 in Array Device Slot 16, SAS Slot: 1+ phys at slot 16
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed674bd
lo0: link state changed to UP
GEOM_MULTIPATH: disk7 created
GEOM_MULTIPATH: da22 added to disk7
GEOM_MULTIPATH: da22 is now active path in disk7
GEOM_MULTIPATH: disk13 created
GEOM_MULTIPATH: da28 added to disk13
GEOM_MULTIPATH: da28 is now active path in disk13
GEOM_MULTIPATH: disk16 created
GEOM_MULTIPATH: da31 added to disk16
GEOM_MULTIPATH: da31 is now active path in disk16
GEOM_MULTIPATH: disk1 created
GEOM_MULTIPATH: da16 added to disk1
GEOM_MULTIPATH: da16 is now active path in disk1
GEOM_MULTIPATH: disk10 created
GEOM_MULTIPATH: da25 added to disk10
GEOM_MULTIPATH: da25 is now active path in disk10
GEOM_MULTIPATH: disk11 created
GEOM_MULTIPATH: da10 added to disk11
GEOM_MULTIPATH: da10 is now active path in disk11
GEOM_MULTIPATH: disk4 created
GEOM_MULTIPATH: da19 added to disk4
GEOM_MULTIPATH: da19 is now active path in disk4
GEOM_MULTIPATH: disk8 created
GEOM_MULTIPATH: da7 added to disk8
GEOM_MULTIPATH: da7 is now active path in disk8
GEOM_MULTIPATH: da12 added to disk13
GEOM_MULTIPATH: disk6 created
GEOM_MULTIPATH: da21 added to disk6
GEOM_MULTIPATH: da21 is now active path in disk6
GEOM_MULTIPATH: disk9 created
GEOM_MULTIPATH: da24 added to disk9
GEOM_MULTIPATH: da24 is now active path in disk9
GEOM_MULTIPATH: disk14 created
GEOM_MULTIPATH: da29 added to disk14
GEOM_MULTIPATH: da29 is now active path in disk14
GEOM_MULTIPATH: da13 added to disk14
GEOM_MULTIPATH: disk15 created
GEOM_MULTIPATH: da30 added to disk15
GEOM_MULTIPATH: da30 is now active path in disk15
GEOM_MULTIPATH: da23 added to disk8
GEOM_MULTIPATH: da26 added to disk11
GEOM_MULTIPATH: disk12 created
GEOM_MULTIPATH: da27 added to disk12
GEOM_MULTIPATH: da27 is now active path in disk12
GEOM_MULTIPATH: disk5 created
GEOM_MULTIPATH: da20 added to disk5
GEOM_MULTIPATH: da20 is now active path in disk5
GEOM_MULTIPATH: disk3 created
GEOM_MULTIPATH: da18 added to disk3
GEOM_MULTIPATH: da18 is now active path in disk3
GEOM_MULTIPATH: da6 added to disk7
GEOM_MULTIPATH: disk2 created
GEOM_MULTIPATH: da17 added to disk2
GEOM_MULTIPATH: da17 is now active path in disk2
GEOM_MULTIPATH: da8 added to disk9
GEOM_MULTIPATH: da5 added to disk6
GEOM_MULTIPATH: da14 added to disk15
GEOM_MULTIPATH: da15 added to disk16
GEOM_MULTIPATH: da9 added to disk10
GEOM_MULTIPATH: da11 added to disk12
GEOM_MULTIPATH: da3 added to disk4
GEOM_MULTIPATH: da1 added to disk2
GEOM_MULTIPATH: da4 added to disk5
GEOM_MULTIPATH: da2 added to disk3
GEOM_MULTIPATH: da0 added to disk1
CPU: Intel(R) Xeon(R) CPU           X5687  @ 3.60GHz (3598.98-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features3=0x9c000000<IBPB,STIBP,L1DFL,SSBD>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
bce2: link state chan


Here are the results from sesutil status and sesutil map commands:

pastebin.com

Last login: Fri Jan 13 17:40:49 on pts/2FreeBSD 13.1-RELEASE-p2 n245412-484f03 - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
pastebin.com pastebin.com

I will look into weather or not those cards are counterfeit next. Any help is appreciated. Thanks.
 
HoneyBadger

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
Super7800 said:
Installed the two new CPUs, and the disks still arent showing up. However, the numbering is all off now. And come to think of it, how could it have been a cpu issue? all the disks showed up fine during install and in the LSI configurator. Its only once installed that they are missing.

Here is DMESG:
Code:
Last login: Fri Jan 13 17:35:03 on pts/0
FreeBSD 13.1-RELEASE-p2 n245412-484f039b1d0 TRUENAS

        TrueNAS (c) 2009-2022, iXsystems, Inc.
        All rights reserved.
        TrueNAS code is released under the modified BSD license with some
        files copyrighted by (c) iXsystems, Inc.

        For more information, documentation, help or support, go here:
        http://truenas.com
Welcome to TrueNAS

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@truenas[~]# dmesg
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.1-RELEASE-p2 n245412-484f039b1d0 TRUENAS amd64
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU           X5687  @ 3.60GHz (3599.06-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 94489280512 (90112 MB)
avail memory = 91937742848 (87678 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <HP     ProLiant>
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 hardware threads
random: unblocking device.
Firmware Warning (ACPI): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20201113/tbfadt-850)
Firmware Warning (ACPI): Invalid length for FADT/Pm2ControlBlock: 32, using default 8 (20201113/tbfadt-850)
ioapic1 <Version 2.0> irqs 24-47
ioapic0 <Version 2.0> irqs 0-23
Launching APs: 1 3 2 5 6 4 14 13 8 9 10 7 11 15 12
random: entropy device external interface
kbd1 at kbdmux0
vtvga0: <VT VGA driver>
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
padlock0: No ACE support.
acpi0: <HP ProLiant>
acpi0: Power Button (fixed)
acpi0: _OSC failed: AE_BUFFER_OVERFLOW
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
apei0: <ACPI Platform Error Interface> on acpi0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci5: <ACPI PCI bus> on pcib5
mps0: <Avago Technologies (LSI) SAS2008> port 0x6000-0x60ff mem 0xfbff0000-0xfbff3fff,0xfbf80000-0xfbfbffff irq 26 at device 0.0 on pci5
mps0: Firmware: 13.00.57.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib6: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci7: <ACPI PCI bus> on pcib7
mps1: <Avago Technologies (LSI) SAS2008> port 0x5000-0x50ff mem 0xfbef0000-0xfbefffff,0xfbe80000-0xfbebffff irq 30 at device 0.0 on pci7
mps1: Firmware: 07.15.08.00, Driver: 21.02.00.00-fbsd
mps1: IOCCapabilities: 185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR>
pcib8: <ACPI PCI-PCI bridge> at device 8.0 on pci0
pci8: <ACPI PCI bus> on pcib8
ix0: <Intel(R) X520 82599ES (SFI/SFP+)> port 0x4000-0x401f mem 0xfbd80000-0xfbdfffff,0xfbd70000-0xfbd73fff irq 31 at device 0.0 on pci8
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 8 RX queues 8 TX queues
ix0: Using MSI-X interrupts with 9 vectors
ix0: allocated for 8 queues
ix0: allocated for 8 rx queues
ix0: Ethernet address: 90:e2:ba:30:6c:78
ix0: PCI Express Bus: Speed 5.0GT/s Width x4
ix0: eTrack 0x0001546c PHY FW V65535
ix1: <Intel(R) X520 82599ES (SFI/SFP+)> port 0x4020-0x403f mem 0xfbc80000-0xfbcfffff,0xfbc70000-0xfbc73fff irq 39 at device 0.1 on pci8
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 8 RX queues 8 TX queues
ix1: Using MSI-X interrupts with 9 vectors
ix1: allocated for 8 queues
ix1: allocated for 8 rx queues
ix1: Ethernet address: 90:e2:ba:30:6c:79
ix1: PCI Express Bus: Speed 5.0GT/s Width x4
ix1: eTrack 0x0001546c PHY FW V65535
pcib9: <ACPI PCI-PCI bridge> at device 9.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 10.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached)
pcib11: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci11: <ACPI PCI bus> on pcib11
bce0: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci11
miibus0: <MII bus> on bce0
brgphy0: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce0: Using defaults for TSO: 65518/35/2048
bce0: Ethernet address: 80:c1:6e:23:a2:46
bce0: ASIC (0x57092003); Rev (C0);
bce0: link state changed to DOWN
Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12)
Coal (RX:6,6,18,18; TX:20,20,80,80)
bce1: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf2000000-0xf3ffffff irq 17 at device 0.1 on pci11
miibus1: <MII bus> on bce1
brgphy1: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce1: Using defaults for TSO: 65518/35/2048
bce1: Ethernet address: 80:c1:6e:23:a2:48
bce1: ASIC (0x57092003);
bce1: link state changed to DOWN
Rev (C0); Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib12: <ACPI PCI-PCI bridge> at device 28.2 on pci0
pci12: <ACPI PCI bus> on pcib12
bce2: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf8000000-0xf9ffffff irq 18 at device 0.0 on pci12
miibus2: <MII bus> on bce2
brgphy2: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus2
brgphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce2: Using defaults for TSO: 65518/35/2048
bce2: Ethernet address: 80:c1:6e:23:a2:4a
bce2: ASIC (0x57092003); Rev (C0); Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI)
Coal (RX:6,6,18,18; TX:20,20,80,80)
bce3: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf6000000-0xf7ffffff irq 19 at device 0.1 on pci12
miibus3: <MII bus> on bce3
brgphy3: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus3
brgphy3:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce3: Using defaults for TSO: 65518/35/2048
bce3: Ethernet address: 80:c1:6e:23:a2:4c
bce3: ASIC (0x57092003); Rev (C0); Bus (PCIe x2, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib13: <ACPI PCI-PCI bridge> at device 28.4 on pci0
pci13: <ACPI PCI bus> on pcib13
uhci0: <HP iLO Standard Virtual USB controller> port 0x3c00-0x3c1f irq 17 at device 0.4 on pci13
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1: <Intel 82801JI (ICH10) USB controller USB-A> port 0x1000-0x101f irq 20 at device 29.0 on pci0
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2: <Intel 82801JI (ICH10) USB controller USB-B> port 0x1020-0x103f irq 23 at device 29.1 on pci0
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
uhci3: <Intel 82801JI (ICH10) USB controller USB-C> port 0x1040-0x105f irq 22 at device 29.2 on pci0
usbus3 on uhci3
usbus3: 12Mbps Full Speed USB v1.0
uhci4: <Intel 82801JI (ICH10) USB controller USB-F> port 0x1060-0x107f irq 23 at device 29.3 on pci0
usbus4 on uhci4
usbus4: 12Mbps Full Speed USB v1.0
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0xf1bf0000-0xf1bf03ff irq 20 at device 29.7 on pci0
usbus5: EHCI version 1.0
usbus5 on ehci0
usbus5: 480Mbps High Speed USB v2.0
pcib14: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci14: <ACPI PCI bus> on pcib14
vgapci0: <VGA-compatible display> port 0x2000-0x20ff mem 0xe8000000-0xefffffff,0xf1cf0000-0xf1cfffff irq 23 at device 3.0 on pci14
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_tz0: <Thermal Zone> on acpi0
ipmi0: <IPMI System Interface> port 0xca2-0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0.
psm0: model IntelliMouse Explorer, device ID 4
acpi_syscontainer0: <System Container> port 0x2e-0x2f,0x620-0x65f,0x680-0x69f,0x600-0x61f,0x660-0x67f,0x300-0x31f on acpi0
uart0: <Non-standard ns8250 class UART with FIFOs> port 0x3f8-0x3ff irq 4 flags0x10 on acpi0
ichwd0: <Intel ICH10 watchdog timer> on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
ichwd0: <Intel ICH10 watchdog timer> at port 0x930-0x937,0x960-0x97f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff pnpid ORM0000 on isa0
uart1: <Non-standard ns8250 class UART with FIFOs> at port 0x2f8 irq 3 on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est0 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est1 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est2 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est3 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est4 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est5 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est6 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est7 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est8 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est9 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est10 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est11 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est12 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est13 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est14 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1c
device_attach: est15 attach returned 6
Timecounter "TSC-low" frequency 1799488487 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ugen1.1: <Intel UHCI root HUB> at usbus1
ugen3.1: <Intel UHCI root HUB> at usbus3
ugen2.1: <Intel UHCI root HUB> at usbus2
ugen4.1: <Intel UHCI root HUB> at usbus4
ugen5.1: <Intel EHCI root HUB> at usbus5
ugen0.1: <HP UHCI root HUB> at usbus0
uhub0 on usbus1
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
uhub1 on usbus3
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
uhub2 on usbus4
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
uhub3 on usbus5
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
uhub4 on usbus2
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
uhub5 on usbus0
uhub5: <HP UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ipmi0: IPMI device rev. 1, firmware rev. 1.89, version 2.0, device support mask0x8f
ipmi0: Number of channels 1
ipmi0: Attached watchdog
ipmi0: Establishing power cycle handler
Trying to mount root from zfs:boot-pool/ROOT/default []...
uhub0: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
ugen0.2: <HP Virtual Keyboard> at usbus0
ukbd0 on uhub5
ukbd0: <Virtual Keyboard > on usbus0
kbd2 at ukbd0
ums0 on uhub5
ums0: <Virtual Mouse> on usbus0
Root mount waiting for: CAM usbus5
Root mount waiting for: CAM usbus5
uhub3: 8 ports with 8 removable, self powered
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
da0 at mps0 bus 0 scbus0 target 8 lun 0
da0: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number YVJSDU3D
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 2861588MB (5860533168 512 byte sectors)
da2 at mps0 bus 0 scbus0 target 10 lun 0
da2: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number YVKP6T9K
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 2861588MB (5860533168 512 byte sectors)
da3 at mps0 bus 0 scbus0 target 11 lun 0
da3: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number YVKUA2WK
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 2861588MB (5860533168 512 byte sectors)
da1 at mps0 bus 0 scbus0 target 9 lun 0
da1: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number YVKMX11K
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 2861588MB (5860533168 512 byte sectors)
da11 at mps0 bus 0 scbus0 target 19 lun 0
da11: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da11: Serial Number YVKP5EAK
da11: 600.000MB/s transfers
da11: Command Queueing enabled
da11: 2861588MB (5860533168 512 byte sectors)
da4 at mps0 bus 0 scbus0 target 12 lun 0
da4: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number YVKNEURK
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 2861588MB (5860533168 512 byte sectors)
da9 at mps0 bus 0 scbus0 target 17 lun 0
da9: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da9: Serial Number YVJT1HBK
da9: 600.000MB/s transfers
da9: Command Queueing enabled
da9: 2861588MB (5860533168 512 byte sectors)
da5 at mps0 bus 0 scbus0 target 13 lun 0
da5: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number P9G7PRRW
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 2861588MB (5860533168 512 byte sectors)
da14 at mps0 bus 0 scbus0 target 22 lun 0
da14: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da14: Serial Number YVKP6UGK
da14: 600.000MB/s transfers
da14: Command Queueing enabled
da14: 2861588MB (5860533168 512 byte sectors)
da15 at mps0 bus 0 scbus0 target 23 lun 0
da15: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da15: Serial Number YVKTY7GK
da15: 600.000MB/s transfers
da15: Command Queueing enabled
da15: 2861588MB (5860533168 512 byte sectors)
da8 at mps0 bus 0 scbus0 target 16 lun 0
da8: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da8: Serial Number P9J262LW
da8: 600.000MB/s transfers
da8: Command Queueing enabled
da8: 2861588MB (5860533168 512 byte sectors)
da6 at mps0 bus 0 scbus0 target 14 lun 0
da6: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number P9HN7B4W
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 2861588MB (5860533168 512 byte sectors)
da13 at mps0 bus 0 scbus0 target 21 lun 0
da13: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da13: Serial Number YVJJ3LMD
da13: 600.000MB/s transfers
da13: Command Queueing enabled
da13: 2861588MB (5860533168 512 byte sectors)
da17 at mps0 bus 0 scbus0 target 46 lun 0
da17: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da17: Serial Number YVKMX11K
da17: 600.000MB/s transfers
da17: Command Queueing enabled
da17: 2861588MB (5860533168 512 byte sectors)
da18 at mps0 bus 0 scbus0 target 47 lun 0
da18: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da18: Serial Number YVKP6T9K
da18: 600.000MB/s transfers
da18: Command Queueing enabled
da18: 2861588MB (5860533168 512 byte sectors)
da20 at mps0 bus 0 scbus0 target 49 lun 0
da20: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da20: Serial Number YVKNEURK
da20: 600.000MB/s transfers
da20: Command Queueing enabled
da20: 2861588MB (5860533168 512 byte sectors)
da23 at mps0 bus 0 scbus0 target 52 lun 0
da23: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da23: Serial Number YVKTYMZK
da23: 600.000MB/s transfers
da23: Command Queueing enabled
da23: 2861588MB (5860533168 512 byte sectors)
da27 at mps0 bus 0 scbus0 target 56 lun 0
da27: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da27: Serial Number YVKP5EAK
da27: 600.000MB/s transfers
da27: Command Queueing enabled
da27: 2861588MB (5860533168 512 byte sectors)
da7 at mps0 bus 0 scbus0 target 15 lun 0
da7: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number YVKTYMZK
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 2861588MB (5860533168 512 byte sectors)
da26 at mps0 bus 0 scbus0 target 55 lun 0
da26: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da26: Serial Number YVKVU0KK
da26: 600.000MB/s transfers
da26: Command Queueing enabled
da26: 2861588MB (5860533168 512 byte sectors)
da30 at mps0 bus 0 scbus0 target 59 lun 0
da30: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da30: Serial Number YVKP6UGK
da30: 600.000MB/s transfers
da30: Command Queueing enabled
da30: 2861588MB (5860533168 512 byte sectors)
da29 at mps0 bus 0 scbus0 target 58 lun 0
da29: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da29: Serial Number YVJJ3LMD
da29: 600.000MB/s transfers
da29: Command Queueing enabled
da29: 2861588MB (5860533168 512 byte sectors)
da12 at mps0 bus 0 scbus0 target 20 lun 0
da12: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da12: Serial Number P9HMRN0W
da12: 600.000MB/s transfers
da12: Command Queueing enabled
da12: 2861588MB (5860533168 512 byte sectors)
da19 at mps0 bus 0 scbus0 target 48 lun 0
da19: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da19: Serial Number YVKUA2WK
da19: 600.000MB/s transfers
da19: Command Queueing enabled
da19: 2861588MB (5860533168 512 byte sectors)
da24 at mps0 bus 0 scbus0 target 53 lun 0
da24: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da24: Serial Number P9J262LW
da24: 600.000MB/s transfers
da24: Command Queueing enabled
da24: 2861588MB (5860533168 512 byte sectors)
da16 at mps0 bus 0 scbus0 target 45 lun 0
da16: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da16: Serial Number YVJSDU3D
da16: 600.000MB/s transfers
da16: Command Queueing enabled
da16: 2861588MB (5860533168 512 byte sectors)
da21 at mps0 bus 0 scbus0 target 50 lun 0
da21: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da21: Serial Number P9G7PRRW
da21: 600.000MB/s transfers
da21: Command Queueing enabled
da21: 2861588MB (5860533168 512 byte sectors)
da34 at mps1 bus 0 scbus1 target 9 lun 0
da34: <HP DG0146FAMWL HPDC> Fixed Direct Access SPC-3 SCSI device
da34: Serial Number 3SD13EP400009005J5LK
da34: 600.000MB/s transfers
da34: Command Queueing enabled
da34: 140014MB (286749488 512 byte sectors)
da10 at mps0 bus 0 scbus0 target 18 lun 0
da10: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da10: Serial Number YVKVU0KK
da10: 600.000MB/s transfers
da10: Command Queueing enabled
da10: 2861588MB (5860533168 512 byte sectors)
da25 at mps0 bus 0 scbus0 target 54 lun 0
da25: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da25: Serial Number YVJT1HBK
da25: 600.000MB/s transfers
da25: Command Queueing enabled
da25: 2861588MB (5860533168 512 byte sectors)
ses0 at mps0 bus 0 scbus0 target 44 lun 0
ses0: <Promise J630s 060=> Fixed Enclosure Services SPC-4 SCSI device
ses0: Serial Number TB0038052886
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SES Device
da35 at mps1 bus 0 scbus1 target 10 lun 0
da35: <HP DG0146FAMWL HPDB> Fixed Direct Access SPC-3 SCSI device
da35: Serial Number 3SD0QESR00009953RW2U
da35: 600.000MB/s transfers
da35: Command Queueing enabled
da35: 140014MB (286749488 512 byte sectors)
da31 at mps0 bus 0 scbus0 target 60 lun 0
da31: <HITACHI HUS723030ALS640 A222> Fixed Direct Access SPC-4 SCSI device
da31: Serial Number YVKTY7GK
da31: 600.000MB/s transfers
da31: Command Queueing enabled
da31: 2861588MB (5860533168 512 byte sectors)
da22 at mps0 bus 0 scbus0 target 51 lun 0
da22: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da22: Serial Number P9HN7B4W
da22: 600.000MB/s transfers
da22: Command Queueing enabled
da22: 2861588MB (5860533168 512 byte sectors)
da28 at mps0 bus 0 scbus0 target 57 lun 0
da28: <HGST HUS724030ALS640 A124> Fixed Direct Access SPC-4 SCSI device
da28: Serial Number P9HMRN0W
da28: 600.000MB/s transfers
da28: Command Queueing enabled
da28: 2861588MB (5860533168 512 byte sectors)
ses1 at mps0 bus 0 scbus0 target 81 lun 0
ses1: <Promise J630s 060=> Fixed Enclosure Services SPC-4 SCSI device
ses1: Serial Number TB0038052886
ses1: 600.000MB/s transfers
ses1: Command Queueing enabled
ses1: SES Device
da32 at mps1 bus 0 scbus1 target 7 lun 0
da32: <HP DG146ABAB4 HPDC> Fixed Direct Access SPC-3 SCSI device
da32: Serial Number 3NM4BY200000982745Y8
da32: 300.000MB/s transfers
da32: Command Queueing enabled
da32: 140014MB (286749488 512 byte sectors)
da33 at mps1 bus 0 scbus1 target 8 lun 0
da33: <HP DG146BB976 HPDC> Fixed Direct Access SPC-3 SCSI device
da33: Serial Number 3NM5606F000098388V18
da33: 300.000MB/s transfers
da33: Command Queueing enabled
da33: 140014MB (286749488 512 byte sectors)
ses0: da0,pass0 in Array Device Slot 1, SAS Slot: 1+ phys at slot 1
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03e9b4e86
ses0: da1,pass1 in Array Device Slot 2, SAS Slot: 1+ phys at slot 2
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecd4b32
ses0: da2,pass2 in Array Device Slot 3, SAS Slot: 1+ phys at slot 3
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecfaf16
ses0: da3,pass3 in Array Device Slot 4, SAS Slot: 1+ phys at slot 4
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed726b6
ses0: da4,pass4 in Array Device Slot 5, SAS Slot: 1+ phys at slot 5
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ece4766
ses0: da5,pass5 in Array Device Slot 6, SAS Slot: 1+ phys at slot 6
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca0580e00ba
ses1: da16,pass17 in Array Device Slot 1, SAS Slot: 1+ phys at slot 1
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03e9b4e85
ses0: da6,pass6 in Array Device Slot 7, SAS Slot: 1+ phys at slot 7
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca0585d2a36
ses1: da17,pass18 in Array Device Slot 2, SAS Slot: 1+ phys at slot 2
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecd4b31
ses0: da7,pass7 in Array Device Slot 8, SAS Slot: 1+ phys at slot 8
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed67ace
ses1: da18,pass19 in Array Device Slot 3, SAS Slot: 1+ phys at slot 3
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecfaf15
ses0: da8,pass8 in Array Device Slot 9, SAS Slot: 1+ phys at slot 9
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca05874ba5a
ses1: da19,pass20 in Array Device Slot 4, SAS Slot: 1+ phys at slot 4
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed726b5
ses0: da9,pass9 in Array Device Slot 10, SAS Slot: 1+ phys at slot 10
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03e9c761e
ses1: da20,pass21 in Array Device Slot 5, SAS Slot: 1+ phys at slot 5
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ece4765
ses0: da10,pass10 in Array Device Slot 11, SAS Slot: 1+ phys at slot 11
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed9d84e
ses1: da21,pass22 in Array Device Slot 6, SAS Slot: 1+ phys at slot 6
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca0580e00b9
ses0: da11,pass11 in Array Device Slot 12, SAS Slot: 1+ phys at slot 12
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecf9ac2
ses1: da22,pass23 in Array Device Slot 7, SAS Slot: 1+ phys at slot 7
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca0585d2a35
ses0: da12,pass12 in Array Device Slot 13, SAS Slot: 1+ phys at slot 13
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca0585c3e42
ses1: da23,pass24 in Array Device Slot 8, SAS Slot: 1+ phys at slot 8
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed67acd
ses0: da13,pass13 in Array Device Slot 14, SAS Slot: 1+ phys at slot 14
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03e8e09da
ses1: da24,pass25 in Array Device Slot 9, SAS Slot: 1+ phys at slot 9
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca05874ba59
ses0: da14,pass14 in Array Device Slot 15, SAS Slot: 1+ phys at slot 15
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ecfafa6
ses1: da25,pass26 in Array Device Slot 10, SAS Slot: 1+ phys at slot 10
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03e9c761d
ses0: da15,pass15 in Array Device Slot 16, SAS Slot: 1+ phys at slot 16
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: parent 500015556694d33f addr 5000cca03ed674be
ses1: da26,pass27 in Array Device Slot 11, SAS Slot: 1+ phys at slot 11
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed9d84d
ses1: da27,pass28 in Array Device Slot 12, SAS Slot: 1+ phys at slot 12
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecf9ac1
ses1: da28,pass29 in Array Device Slot 13, SAS Slot: 1+ phys at slot 13
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca0585c3e41
ses1: da29,pass30 in Array Device Slot 14, SAS Slot: 1+ phys at slot 14
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03e8e09d9
ses1: da30,pass31 in Array Device Slot 15, SAS Slot: 1+ phys at slot 15
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ecfafa5
ses1: da31,pass32 in Array Device Slot 16, SAS Slot: 1+ phys at slot 16
ses1:  phy 0: SAS device type 1 phy 0 Target ( SSP )
ses1:  phy 0: parent 500015556694d23f addr 5000cca03ed674bd
lo0: link state changed to UP
GEOM_MULTIPATH: disk7 created
GEOM_MULTIPATH: da22 added to disk7
GEOM_MULTIPATH: da22 is now active path in disk7
GEOM_MULTIPATH: disk13 created
GEOM_MULTIPATH: da28 added to disk13
GEOM_MULTIPATH: da28 is now active path in disk13
GEOM_MULTIPATH: disk16 created
GEOM_MULTIPATH: da31 added to disk16
GEOM_MULTIPATH: da31 is now active path in disk16
GEOM_MULTIPATH: disk1 created
GEOM_MULTIPATH: da16 added to disk1
GEOM_MULTIPATH: da16 is now active path in disk1
GEOM_MULTIPATH: disk10 created
GEOM_MULTIPATH: da25 added to disk10
GEOM_MULTIPATH: da25 is now active path in disk10
GEOM_MULTIPATH: disk11 created
GEOM_MULTIPATH: da10 added to disk11
GEOM_MULTIPATH: da10 is now active path in disk11
GEOM_MULTIPATH: disk4 created
GEOM_MULTIPATH: da19 added to disk4
GEOM_MULTIPATH: da19 is now active path in disk4
GEOM_MULTIPATH: disk8 created
GEOM_MULTIPATH: da7 added to disk8
GEOM_MULTIPATH: da7 is now active path in disk8
GEOM_MULTIPATH: da12 added to disk13
GEOM_MULTIPATH: disk6 created
GEOM_MULTIPATH: da21 added to disk6
GEOM_MULTIPATH: da21 is now active path in disk6
GEOM_MULTIPATH: disk9 created
GEOM_MULTIPATH: da24 added to disk9
GEOM_MULTIPATH: da24 is now active path in disk9
GEOM_MULTIPATH: disk14 created
GEOM_MULTIPATH: da29 added to disk14
GEOM_MULTIPATH: da29 is now active path in disk14
GEOM_MULTIPATH: da13 added to disk14
GEOM_MULTIPATH: disk15 created
GEOM_MULTIPATH: da30 added to disk15
GEOM_MULTIPATH: da30 is now active path in disk15
GEOM_MULTIPATH: da23 added to disk8
GEOM_MULTIPATH: da26 added to disk11
GEOM_MULTIPATH: disk12 created
GEOM_MULTIPATH: da27 added to disk12
GEOM_MULTIPATH: da27 is now active path in disk12
GEOM_MULTIPATH: disk5 created
GEOM_MULTIPATH: da20 added to disk5
GEOM_MULTIPATH: da20 is now active path in disk5
GEOM_MULTIPATH: disk3 created
GEOM_MULTIPATH: da18 added to disk3
GEOM_MULTIPATH: da18 is now active path in disk3
GEOM_MULTIPATH: da6 added to disk7
GEOM_MULTIPATH: disk2 created
GEOM_MULTIPATH: da17 added to disk2
GEOM_MULTIPATH: da17 is now active path in disk2
GEOM_MULTIPATH: da8 added to disk9
GEOM_MULTIPATH: da5 added to disk6
GEOM_MULTIPATH: da14 added to disk15
GEOM_MULTIPATH: da15 added to disk16
GEOM_MULTIPATH: da9 added to disk10
GEOM_MULTIPATH: da11 added to disk12
GEOM_MULTIPATH: da3 added to disk4
GEOM_MULTIPATH: da1 added to disk2
GEOM_MULTIPATH: da4 added to disk5
GEOM_MULTIPATH: da2 added to disk3
GEOM_MULTIPATH: da0 added to disk1
CPU: Intel(R) Xeon(R) CPU           X5687  @ 3.60GHz (3598.98-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features3=0x9c000000<IBPB,STIBP,L1DFL,SSBD>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
bce2: link state chan


Here are the results from sesutil status and sesutil map commands:

pastebin.com

Last login: Fri Jan 13 17:40:49 on pts/2FreeBSD 13.1-RELEASE-p2 n245412-484f03 - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
pastebin.com pastebin.com

I will look into weather or not those cards are counterfeit next. Any help is appreciated. Thanks.
Click to expand...

I would still suspect there is an issue with your second card being detected. Your dmesg output still shows only two HBAs (mps0/mps1) and two enclosures in both dmesg and sesutil map (ses0/ses1) - I would expect to see three of each, given that there are three HBAs physically installed and three SES managable enclosures.

Checking to see if the HBA connected to the second/unrecognized shelf works in a different machine or different slot would be a good troubleshooting step.

You are also seeing "all the drives" in the BIOS as it hasn't merged the two SAS paths into a single device yet. Once this works, you should expect to see 32x2 = 64 "drives" in the LSI BIOS screen, and 32 drives in the TrueNAS UI.
 
S

Super7800

Dabbler
Joined
Feb 2, 2021
Messages
16
Ok I will try swapping them/using them one by one. I don't know if I believe that (but I am definitely not knowledgeable in the topic) since during truenas install all the drives show up "correctly", and the 4 boot drives show up as 4 not 8. But again I don't know.

Here are detailed pics of the two cards. At first glance they don't seem counterfit. But I don't know and will investigate further. https://postimg.cc/gallery/Sks1LBN
 
You must log in or register to reply here.

Similar threads

S
TrueNAS showing one disk, while gome disk list shows two.
Replies
0
Views
642
saurabhperiwal
S
W
Mirrored boot device - Only one showing in disk list
Replies
1
Views
1K
Bikerchris
Bikerchris
Stefan042
LSI question
Replies
11
Views
2K
jgreco
jgreco
A
Truenas EFI booting error fresh install
Replies
0
Views
2K
AVSION
A
C
TrueNAS only seeing one disk per LSI slot
Replies
9
Views
1K
CJRoss
C
Top