FreeNAS Crashing on Fresh Install

[awildeone]

Hi all,

Been running into an issue over the last week and am running out of ideas. I was previously using Unraid on trial and found the performance was bad. So I tried setting up a FreeNAS 11.3 instance last weekend but have had problems for a week. If the server was not being accessed, it would take about one hour to crash and start producing similar messages to the one below. This morning nuked everything and started over with a fresh install of 11.2 hoping it was just a version issue. Now the issue comes up within 15 minutes of powering on. When this happens I lose access to the WebUI and all services and the message below appears when running dmesg. I have an online and local backup from when I was running UnRaid so I can nuke and start over if needed.

What is unavailable when it crashes:

• Plugins/Jails
• File Sharing
• WebUI

Below is the error output from the console when the system crashes. SMART reports do not list any immediate errors but since the messages below are related to the hard drives, I imagine that is going to play a part. I have SSH turned on so I am able to still access that for logs and debugging. I reseated all the hard drives this morning to make sure the connection was solid. I still have my Dell PowerVault MD1200 that I can test with if it seems that the array itself is possibly the issue. I also have iLO setup in case that would be another tool to use in troubleshooting. iLO is how I do all my setup since I run my server completely headless (and I don't have a VGA monitor).

Hardware:

• HP Proliant DL360 G7
• CPU: Dual Intel Xeon E5620 2.40GHz
• RAM: 12GB 10600R ECC DDR3
• Boot Disk: Adata 16GB USB 3.1 (Connected to internal 2.0 port).
• HBA Card: LSI/Avago 9207-8e (IT Mode)
• SAS Enclosure: Sans Digital EliteSTOR ES316X6+BS (16-Bay Disk Shelf)
• Hard Drives (RAIDz2 w/ Cache):
  • 2x WD RED 1TB (2-3 y/o, power on time about 1 year)
  • 2x WD Blue 1TB (Brand New)
  • 1x WD Green 1TB
  • 1x SanDisk 256GB SSD

Code:

Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.2-STABLE #0 r325575+c9231c7d6bd(HEAD): Mon Nov 18 22:46:47 UTC 2019
    root@nemesis:/freenas-releng/freenas/_BE/objs/freenas-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
VT(vga): resolution 640x480
CPU: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz (2400.14-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 12884901888 (12288 MB)
avail memory = 12416528384 (11841 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <HP     ProLiant>
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 hardware threads
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
Firmware Warning (ACPI): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20171214/tbfadt-850)
Firmware Warning (ACPI): Invalid length for FADT/Pm2ControlBlock: 32, using default 8 (20171214/tbfadt-850)
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic0 <Version 2.0> irqs 0-23 on motherboard
SMP: AP CPU #1 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #4 Launched!
Timecounter "TSC-low" frequency 1200070494 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
padlock0: No ACE support.
acpi0: <HP ProLiant> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
cpu8: <ACPI CPU> on acpi0
cpu9: <ACPI CPU> on acpi0
cpu10: <ACPI CPU> on acpi0
cpu11: <ACPI CPU> on acpi0
cpu12: <ACPI CPU> on acpi0
cpu13: <ACPI CPU> on acpi0
cpu14: <ACPI CPU> on acpi0
cpu15: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
ciss0: <HP Smart Array P410i> port 0x4000-0x40ff mem 0xfbc00000-0xfbdfffff,0xfbbf0000-0xfbbf0fff irq 28 at device 0.0 on pci1
ciss0: PERFORMANT Transport
pcib2: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci3: <ACPI PCI bus> on pcib3
mps0: <Avago Technologies (LSI) SAS2308> port 0x5000-0x50ff mem 0xfbff0000-0xfbffffff,0xfbf80000-0xfbfbffff irq 24 at device 0.0 on pci3
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
pcib4: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci7: <ACPI PCI bus> on pcib7
bce0: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf6000000-0xf7ffffff irq 30 at device 0.0 on pci7
miibus0: <MII bus> on bce0
brgphy0: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce0: Using defaults for TSO: 65518/35/2048
bce0: Ethernet address: b4:b5:2f:69:4f:84
bce0:
bce0: link state changed to DOWN
ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12)
Coal (RX:6,6,18,18; TX:20,20,80,80)
bce1: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf4000000-0xf5ffffff irq 37 at device 0.1 on pci7
miibus1: <MII bus> on bce1
brgphy1: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce1: Using defaults for TSO: 65518/35/2048
bce1: Ethernet address: b4:b5:2f:69:4f:86
bce1:
bce1: link state changed to DOWN
ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib8: <ACPI PCI-PCI bridge> at device 8.0 on pci0
pci8: <ACPI PCI bus> on pcib8
bce2: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf8000000-0xf9ffffff irq 31 at device 0.0 on pci8
miibus2: <MII bus> on bce2
brgphy2: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus2
brgphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce2: Using defaults for TSO: 65518/35/2048
bce2: Ethernet address: b4:b5:2f:69:3f:c8
bce2:
bce2: link state changed to DOWN
ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib9: <ACPI PCI-PCI bridge> at device 9.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 10.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached)
pcib11: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci11: <ACPI PCI bus> on pcib11
pcib12: <ACPI PCI-PCI bridge> at device 28.4 on pci0
pci12: <ACPI PCI bus> on pcib12
uhci0: <HP iLO Standard Virtual USB controller> port 0x3c00-0x3c1f irq 17 at device 0.4 on pci12
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1: <Intel 82801JI (ICH10) USB controller USB-A> port 0x1000-0x101f irq 20 at device 29.0 on pci0
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2: <Intel 82801JI (ICH10) USB controller USB-B> port 0x1020-0x103f irq 23 at device 29.1 on pci0
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
uhci3: <Intel 82801JI (ICH10) USB controller USB-C> port 0x1040-0x105f irq 22 at device 29.2 on pci0
usbus3 on uhci3
usbus3: 12Mbps Full Speed USB v1.0
uhci4: <Intel 82801JI (ICH10) USB controller USB-F> port 0x1060-0x107f irq 23 at device 29.3 on pci0
usbus4 on uhci4
usbus4: 12Mbps Full Speed USB v1.0
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0xf3bf0000-0xf3bf03ff irq 20 at device 29.7 on pci0
usbus5: EHCI version 1.0
usbus5 on ehci0
usbus5: 480Mbps High Speed USB v2.0
pcib13: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci13: <ACPI PCI bus> on pcib13
vgapci0: <VGA-compatible display> port 0x2000-0x20ff mem 0xe8000000-0xefffffff,0xf3cf0000-0xf3cfffff irq 23 at device 3.0 on pci13
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH10 SATA300 controller> at device 31.2 on pci0
ata2: <ATA channel> at channel 0 on atapci0
acpi_tz0: <Thermal Zone> on acpi0
ipmi0: <IPMI System Interface> port 0xca2-0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
acpi_syscontainer0: <System Container> port 0x2e-0x2f,0x620-0x65f,0x680-0x69f,0x600-0x61f,0x660-0x67f,0x300-0x31f on acpi0
uart0: <Non-standard ns8250 class UART with FIFOs> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
ichwd0: <Intel ICH10 watchdog timer> on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
ichwd0: <Intel ICH10 watchdog timer> at port 0x930-0x937,0x960-0x97f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0
uart1: <Non-standard ns8250 class UART with FIFOs> at port 0x2f8 irq 3 on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est0 attach returned 6
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est1 attach returned 6
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est2 attach returned 6
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est3 attach returned 6
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est4 attach returned 6
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est5 attach returned 6
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est6 attach returned 6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est7 attach returned 6
coretemp8: <CPU On-Die Thermal Sensors> on cpu8
est8: <Enhanced SpeedStep Frequency Control> on cpu8
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est8 attach returned 6
coretemp9: <CPU On-Die Thermal Sensors> on cpu9
est9: <Enhanced SpeedStep Frequency Control> on cpu9
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est9 attach returned 6
coretemp10: <CPU On-Die Thermal Sensors> on cpu10
est10: <Enhanced SpeedStep Frequency Control> on cpu10
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est10 attach returned 6
coretemp11: <CPU On-Die Thermal Sensors> on cpu11
est11: <Enhanced SpeedStep Frequency Control> on cpu11
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est11 attach returned 6
coretemp12: <CPU On-Die Thermal Sensors> on cpu12
est12: <Enhanced SpeedStep Frequency Control> on cpu12
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est12 attach returned 6
coretemp13: <CPU On-Die Thermal Sensors> on cpu13
est13: <Enhanced SpeedStep Frequency Control> on cpu13
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est13 attach returned 6
coretemp14: <CPU On-Die Thermal Sensors> on cpu14
est14: <Enhanced SpeedStep Frequency Control> on cpu14
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est14 attach returned 6
coretemp15: <CPU On-Die Thermal Sensors> on cpu15
est15: <Enhanced SpeedStep Frequency Control> on cpu15
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 13
device_attach: est15 attach returned 6
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
freenas_sysctl: adding account.
freenas_sysctl: adding directoryservice.
freenas_sysctl: adding middlewared.
freenas_sysctl: adding network.
freenas_sysctl: adding services.
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ugen3.1: <Intel UHCI root HUB> at usbus3
ugen1.1: <Intel UHCI root HUB> at usbus1
ugen4.1: <Intel UHCI root HUB> at usbus4
ugen2.1: <Intel UHCI root HUB> at usbus2
ugen5.1: <Intel EHCI root HUB> at usbus5
ugen0.1: <HP UHCI root HUB> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
uhub4: <HP UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
mps0: SAS Address for SATA device = 4e66525204c9e17e
mps0: SAS Address for SATA device = 3976383cba9b8d7e
ipmi0: IPMI device rev. 1, firmware rev. 1.28, version 2.0
mps0: SAS Address from SATA device = 4e66525204c9e17e
mps0: SAS Address from SATA device = 3976383cba9b8d7e
mps0: SAS Address for SATA device = dd615e69f4bac689
ipmi0: Number of channels 0
mps0: SAS Address for SATA device = 4e626647f2c7cf95
mps0: SAS Address for SATA device = 6305523a9b71a0a4
ipmi0: Attached watchdog
mps0: SAS Address for SATA device = dd616467f3c9b271
mps0: SAS Address for SATA device = 6269520597baa982
mps0: SAS Address from SATA device = dd615e69f4bac689
mps0: SAS Address from SATA device = 4e626647f2c7cf95
mps0: SAS Address from SATA device = 6305523a9b71a0a4
mps0: SAS Address from SATA device = dd616467f3c9b271
mps0: SAS Address from SATA device = 6269520597baa982
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
ugen0.2: <HP Virtual Keyboard> at usbus0
ukbd0 on uhub4
ukbd0: <Virtual Keyboard > on usbus0
kbd2 at ukbd0
uhub3: 8 ports with 8 removable, self powered
ugen5.2: <Innostor PenDrive> at usbus5
umass0 on uhub3
umass0: <Innostor PenDrive, class 0/0, rev 2.10/0.01, addr 2> on usbus5
umass0:  SCSI over Bulk-Only; quirks = 0x8100
umass0:5:0: Attached to scbus5
random: unblocking device.
ses0 at mps0 bus 0 scbus2 target 12 lun 0
ses0: <SansDigi SDEXP24 0e03> Fixed Enclosure Services SPC-3 SCSI device
ses0: Serial Number
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 ENC Device
da7 at umass-sim0 bus 0 scbus5 target 0 lun 0
da0 at mps0 bus 0 scbus2 target 8 lun 0
da7: <Innostor Innostor 1.00> Removable Direct Access SPC-4 SCSI device
da7: Serial Number 151876716014230
da7: 40.000MB/s transfers
da7: 15074MB (30871552 512 byte sectors)
da7: quirks=0x12<NO_6_BYTE,NO_RC16>
da6 at mps0 bus 0 scbus2 target 24 lun 0
da0: <ATA WDC WD10EFRX-68F 0A82> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number WD-WCC4J4DNXFZA
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da6: <ATA WDC WD10EZEX-08W 1A02> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number WD-WCC6Y5VRRSE2
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da0: 953869MB (1953525168 512 byte sectors)
da0: quirks=0x8<4K>
da6: 953869MB (1953525168 512 byte sectors)
da1 at mps0 bus 0 scbus2 target 10 lun 0
ses0: da0,pass0: Element descriptor: 'Disk-01'
da4 at mps0 bus 0 scbus2 target 20 lun 0
da1: <ATA WDC WD10EFRX-68F 0A82> Fixed Direct Access SPC-4 SCSI device
da4: <ATA SanDisk SDSSDA24 00RL> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number 151554400865       
da4: 600.000MB/s transfers
da4: Command Queueing enabled
ses0: da0,pass0: SAS Device Slot Element: 1 Phys at Slot 1
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500180d0000564ff addr 500180d0000564c0
da1: Serial Number WD-WCC4J0XCFEHX
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 953869MB (1953525168 512 byte sectors)
da4: 228936MB (468862128 512 byte sectors)
da1: quirks=0x8<4K>
da2 at mps0 bus 0 scbus2 target 13 lun 0
da3 at mps0 bus 0 scbus2 target 14 lun 0
da2: <ATA WDC WD10EADS-22M 0A01> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number WD-WCAV5H015044
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 953869MB (1953525168 512 byte sectors)
da5 at mps0 bus 0 scbus2 target 22 lun 0
da3: <ATA APPLE HDD ST1000 AQ03> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number Z4YER3LS
da3: 300.000MB/s transfers
da3: Command Queueing enabled
da3: 953869MB (1953525168 512 byte sectors)
da5: <ATA WDC WD10EZEX-08W 1A02> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number WD-WCC6Y5PTSDYJ
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 953869MB (1953525168 512 byte sectors)
ses0: da2,pass3: Element descriptor: 'Disk-02'
ses0: da2,pass3: SAS Device Slot Element: 1 Phys at Slot 2
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500180d0000564ff addr 500180d0000564c1
ses0: da1,pass1: Element descriptor: 'Disk-05'
ses0: da1,pass1: SAS Device Slot Element: 1 Phys at Slot 5
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500180d0000564ff addr 500180d0000564dc
ses0: da3,pass4: Element descriptor: 'Disk-06'
ses0: da3,pass4: SAS Device Slot Element: 1 Phys at Slot 6
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500180d0000564ff addr 500180d0000564dd
ses0: da6,pass7: Element descriptor: 'Disk-10'
ses0: da6,pass7: SAS Device Slot Element: 1 Phys at Slot 10
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500180d0000564ff addr 500180d0000564e1
ses0: da4,pass5: Element descriptor: 'Disk-12'
ses0: da4,pass5: SAS Device Slot Element: 1 Phys at Slot 12
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500180d0000564ff addr 500180d0000564e3
ses0: da5,pass6: Element descriptor: 'Disk-13'
ses0: da5,pass6: SAS Device Slot Element: 1 Phys at Slot 13
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500180d0000564ff addr 500180d0000564d8
Trying to mount root from zfs:freenas-boot/ROOT/default []...
GEOM_MIRROR: Device mirror/swap0 launched (2/2).
GEOM_MIRROR: Device mirror/swap1 launched (2/2).
GEOM_ELI: Device mirror/swap0.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device mirror/swap1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: hardware
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI> UCP/8/48/0x3f8<EDG,THR,REA,WRI,INV,QUA,PRC> UCF/1/48/0x60<REA,WRI>
bce0: Gigabit link up!
bce0: link state changed to UP
bce0: Gigabit link up!
ums0 on uhub4
ums0: <Virtual Mouse> on usbus0
GEOM_MIRROR: Device mirror/swap2 launched (2/2).
GEOM_ELI: Device mirror/swap2.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: hardware
mps0: mpssas_prepare_remove: Sending reset for target ID 14
da3 at mps0 bus 0 scbus2 target 14 lun 0
da3: <ATA APPLE HDD ST1000 AQ03> s/n Z4YER3LS detached
GEOM_MIRROR: Device swap0: provider da3p1 disconnected.
(da3:mps0:0:14:0): Periph destroyed
mps0: Unfreezing devq for target ID 14
GEOM_ELI: Device mirror/swap0.eli destroyed.
GEOM_MIRROR: Device swap0: provider destroyed.
GEOM_MIRROR: Device swap0 destroyed.
mps0: mpssas_prepare_remove: Sending reset for target ID 8
mps0: mpssas_prepare_remove: Sending reset for target ID 13
da0 at mps0 bus 0 scbus2 target 8 lun 0
da0: <ATA WDC WD10EFRX-68F 0A82> s/n WD-WCC4J4DNXFZA detached
da2 at mps0 bus 0 scbus2 target 13 lun 0
da2: <ATA WDC WD10EADS-22M 0A01> s/n WD-WCAV5H015044 detached
GEOM_MIRROR: Device swap1: provider da2p1 disconnected.
GEOM_MIRROR: Device swap2: provider da0p1 disconnected.
(da2:mps0:0:13:0): Periph destroyed
(da0:mps0:0:8:0): Periph destroyed
mps0: Unfreezing devq for target ID 8
mps0: Unfreezing devq for target ID 13
mps0: mpssas_prepare_remove: Sending reset for target ID 22
mps0: mpssas_prepare_remove: Sending reset for target ID 10
mps0: mpssas_prepare_remove: Sending reset for target ID 24
da5 at mps0 bus 0 scbus2 target 22 lun 0
da5: <ATA WDC WD10EZEX-08W 1A02> s/n WD-WCC6Y5PTSDYJ detached
da1 at mps0 bus 0 scbus2 target 10 lun 0
da1: <ATA WDC WD10EFRX-68F 0A82> s/n WD-WCC4J0XCFEHX detached
da6 at mps0 bus 0 scbus2 target 24 lun 0
da6: <ATA WDC WD10EZEX-08W 1A02> s/n WD-WCC6Y5VRRSE2 detached
GEOM_MIRROR: Device swap1: provider da1p1 disconnected.
GEOM_MIRROR: Device swap2: provider da6p1 disconnected.
GEOM_MIRROR: Device swap2: provider destroyed.
GEOM_MIRROR: Device swap2 destroyed.
GEOM_ELI: Device mirror/swap2.eli destroyed.
GEOM_ELI: Detached mirror/swap2.eli on last close.
(da1:mps0:0:10:0): Periph destroyed
(da6:mps0:0:24:0): Periph destroyed
(da5:mps0:0:22:0): Periph destroyed
GEOM_MIRROR: Device swap1: provider destroyed.
GEOM_MIRROR: Device swap1 destroyed.
mps0: Unfreezing devq for target ID 22
mps0: Unfreezing devq for target ID 10
mps0: Unfreezing devq for target ID 24
GEOM_ELI: Device mirror/swap1.eli destroyed.
GEOM_ELI: Detached mirror/swap1.eli on last close.
mpssas_get_sata_identify: error reading SATA PASSTHRU; iocstatus = 0x47
mpssas_get_sata_identify: error reading SATA PASSTHRU; iocstatus = 0x804b
mps0: Sleeping 3 seconds after SATA ID error to wait for spinup
mps0: SAS Address for SATA device = dd615e69f4bac689
mps0: SAS Address for SATA device = dd616467f3c9b271
mps0: SAS Address from SATA device = dd615e69f4bac689
mps0: SAS Address from SATA device = dd616467f3c9b271
ses0: da0,pass0: Element descriptor: 'Disk-10'
ses0: da0,pass0: SAS Device Slot Element: 1 Phys at Slot 10
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500180d0000564ff addr 500180d0000564e1
ses0: da1,pass1: Element descriptor: 'Disk-13'
ses0: da1,pass1: SAS Device Slot Element: 1 Phys at Slot 13
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500180d0000564ff addr 500180d0000564d8
da0 at mps0 bus 0 scbus2 target 24 lun 0
da0: <ATA WDC WD10EZEX-08W 1A02> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number WD-WCC6Y5VRRSE2
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 953869MB (1953525168 512 byte sectors)
da1 at mps0 bus 0 scbus2 target 22 lun 0
da1: <ATA WDC WD10EZEX-08W 1A02> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number WD-WCC6Y5PTSDYJ
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 953869MB (1953525168 512 byte sectors)
mps0: SAS Address for SATA device = 3976383cba9b8d7e
mps0: SAS Address from SATA device = 3976383cba9b8d7e
ses0: da2,pass3: Element descriptor: 'Disk-02'
ses0: da2,pass3: SAS Device Slot Element: 1 Phys at Slot 2
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500180d0000564ff addr 500180d0000564c1
da2 at mps0 bus 0 scbus2 target 13 lun 0
da2: <ATA WDC WD10EADS-22M 0A01> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number WD-WCAV5H015044
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 953869MB (1953525168 512 byte sectors)
mps0: SAS Address for SATA device = 4e66525204c9e17e
mps0: SAS Address from SATA device = 4e66525204c9e17e
mps0: SAS Address for SATA device = 4e626647f2c7cf95
mps0: SAS Address from SATA device = 4e626647f2c7cf95
ses0: da5,pass6: Element descriptor: 'Disk-01'
ses0: da5,pass6: SAS Device Slot Element: 1 Phys at Slot 1
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500180d0000564ff addr 500180d0000564c0
ses0: da3,pass4: Element descriptor: 'Disk-05'
ses0: da3,pass4: SAS Device Slot Element: 1 Phys at Slot 5
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500180d0000564ff addr 500180d0000564dc
da3 at mps0 bus 0 scbus2 target 10 lun 0
da3: <ATA WDC WD10EFRX-68F 0A82> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number WD-WCC4J0XCFEHX
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 953869MB (1953525168 512 byte sectors)
da3: quirks=0x8<4K>
da5 at mps0 bus 0 scbus2 target 8 lun 0
da5: <ATA WDC WD10EFRX-68F 0A82> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number WD-WCC4J4DNXFZA
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 953869MB (1953525168 512 byte sectors)
da5: quirks=0x8<4K>

 

[awildeone]

tl;dr: I think it was the SAS expander on the JBOD.

So I did manage to solve my issue. Not sure specifically what was wrong, but it looks like the controller on my Sans Digital SAS expander was failing. I replaced it with my previous Dell PowerVault MD1200 and have had no issues in almost 4 weeks. I have two of the Sans Digital units (one had a failed PSU so I swapped the working PSU into the second one. I don't have an extra 5 disks to create a second vDev so I'll find out in the future if that controller works. Also, it could be the PSU with a bad power rail, but that's an adventure for the future.

If everything is awful, its a server case with a 16 bay SATA backplane that I can build an ATX server in the future. Just use all new parts lol. I'll find a way to make lemonade!

 

[subhuman]

Glad you solved it. just as a FYI,

Boot Disk: Adata 16GB USB 3.1 (Connected to internal 2.0 port).

This hasn't been recommended for a while now. It's much preferred to use a SSD or HDD.