one800higgins
Dabbler
- Joined
- Jul 16, 2015
- Messages
- 24
Just ran into a problem and I'm having an issue figuring out what caused it and/or how to resolve it. I rebooted my FreeNAS earlier today and when it came back online, one of my arrays is no longer found:
"The volume vol1 (ZFS) state is UNKNOWN"
vol1 was a 2 SSD array, mirrored. FreeNAS is suddenly acting like the disks don't even exist anymore... My other volumn (6 disk array) is running just fine.
I run regular health checks, scrubs, etc and I've had no errors or warnings on these disks. It's only after rebooting that suddenly this issue has come up and I can't seem to figure out what happened or how to resolve it (if I even can).
Anyone have any suggestions / thoughts?
EDIT: More info below...
Version: FreeNAS-9.10-STABLE-201604261518
CPU: Intel(R) Xeon(R) CPU E3-1231 v3 @ 3.40GHz
Mobo: SuperMicro MBD-X10SLH-F-O
RAM: 2x Crucial 16GB KVR16R11D4/16
Hard drives: 2x SanDisk SDSSDX-240G-G25 (plus 6x 3TB WD Reds, but no issue there)
"zpool status" doesn't recognize the missing array (vol1):
"dmesg" output:
"smartctl --scan" doesn't list the disks:
"The volume vol1 (ZFS) state is UNKNOWN"
vol1 was a 2 SSD array, mirrored. FreeNAS is suddenly acting like the disks don't even exist anymore... My other volumn (6 disk array) is running just fine.
I run regular health checks, scrubs, etc and I've had no errors or warnings on these disks. It's only after rebooting that suddenly this issue has come up and I can't seem to figure out what happened or how to resolve it (if I even can).
Anyone have any suggestions / thoughts?
EDIT: More info below...
Version: FreeNAS-9.10-STABLE-201604261518
CPU: Intel(R) Xeon(R) CPU E3-1231 v3 @ 3.40GHz
Mobo: SuperMicro MBD-X10SLH-F-O
RAM: 2x Crucial 16GB KVR16R11D4/16
Hard drives: 2x SanDisk SDSSDX-240G-G25 (plus 6x 3TB WD Reds, but no issue there)
"zpool status" doesn't recognize the missing array (vol1):
Code:
pool: freenas-boot state: ONLINE scan: scrub repaired 0 in 0h2m with 0 errors on Thu Mar 31 03:47:37 2016 config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da0p2 ONLINE 0 0 0 da1p2 ONLINE 0 0 0 errors: No known data errors pool: vol0 state: ONLINE scan: scrub repaired 0 in 4h44m with 0 errors on Fri Apr 15 08:44:19 2016 config: NAME STATE READ WRITE CKSUM vol0 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/5698b868-3222-11e5-b6bc-0cc47a4ec610 ONLINE 0 0 0 gptid/57686acd-3222-11e5-b6bc-0cc47a4ec610 ONLINE 0 0 0 gptid/57d5ccdb-3222-11e5-b6bc-0cc47a4ec610 ONLINE 0 0 0 gptid/583f4da8-3222-11e5-b6bc-0cc47a4ec610 ONLINE 0 0 0 gptid/58a2d378-3222-11e5-b6bc-0cc47a4ec610 ONLINE 0 0 0 gptid/5901c578-3222-11e5-b6bc-0cc47a4ec610 ONLINE 0 0 0 errors: No known data errors
"dmesg" output:
Code:
Copyright (c) 1992-2016 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.3-RELEASE #0 f935af8(freebsd10): Tue Apr 26 08:30:39 PDT 2016 root@build.ixsystems.com:/tank/home/nightlies/build-freenas9/_BE/objs/tank/home/nightlies/build-freenas9/_BE/trueos/sys/FreeNAS.amd64 amd64 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 CPU: Intel(R) Xeon(R) CPU E3-1231 v3 @ 3.40GHz (3400.07-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306c3 Family=0x6 Model=0x3c Stepping=3 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x21<LAHF,ABM> Structured Extended Features=0x2fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG> XSAVE Features=0x1<XSAVEOPT> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 34896609280 (33280 MB) avail memory = 33261326336 (31720 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <SUPERM SMCI--MB> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 random: <Software, Yarrow> initialized WARNING: VIMAGE (virtualized network stack) is a highly experimental feature. ioapic0 <Version 2.0> irqs 0-23 on motherboard module_register_init: MOD_LOAD (vesa, 0xffffffff80dce260, 0) error 19 kbd1 at kbdmux0 cryptosoft0: <software crypto> on motherboard aesni0: <AES-CBC,AES-XTS> on motherboard padlock0: No ACE support. acpi0: <SUPERM SMCI--MB> on motherboard acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 cpu2: <ACPI CPU> on acpi0 cpu3: <ACPI CPU> on acpi0 cpu4: <ACPI CPU> on acpi0 cpu5: <ACPI CPU> on acpi0 cpu6: <ACPI CPU> on acpi0 cpu7: <ACPI CPU> on acpi0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 550 atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 xhci0: <Intel Lynx Point USB 3.0 controller> mem 0xf7300000-0xf730ffff irq 16 at device 20.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA xhci0: Port routing mask set to 0xffffffff usbus0 on xhci0 pci0: <simple comms> at device 22.0 (no driver attached) pci0: <simple comms> at device 22.1 (no driver attached) ehci0: <Intel Lynx Point USB 2.0 controller USB-B> mem 0xf7314000-0xf73143ff irq 16 at device 26.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci0 pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 pci1: <ACPI PCI bus> on pcib1 pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1 pci2: <ACPI PCI bus> on pcib2 vgapci0: <VGA-compatible display> port 0xe000-0xe07f mem 0xf6000000-0xf6ffffff,0xf7000000-0xf701ffff irq 16 at device 0.0 on pci2 vgapci0: Boot video device pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0 pci3: <ACPI PCI bus> on pcib3 igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xd000-0xd01f mem 0xf7200000-0xf727ffff,0xf7280000-0xf7283fff irq 18 at device 0.0 on pci3 igb0: Using MSIX interrupts with 5 vectors igb0: Ethernet address: 0c:c4:7a:4e:c6:10 igb0: Bound queue 0 to cpu 0 igb0: Bound queue 1 to cpu 1 igb0: Bound queue 2 to cpu 2 igb0: Bound queue 3 to cpu 3 pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0 pci4: <ACPI PCI bus> on pcib4 igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xc000-0xc01f mem 0xf7100000-0xf717ffff,0xf7180000-0xf7183fff irq 19 at device 0.0 on pci4 igb1: Using MSIX interrupts with 5 vectors igb1: Ethernet address: 0c:c4:7a:4e:c6:11 igb1: Bound queue 0 to cpu 4 igb1: Bound queue 1 to cpu 5 igb1: Bound queue 2 to cpu 6 igb1: Bound queue 3 to cpu 7 pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0 pci5: <ACPI PCI bus> on pcib5 ehci1: <Intel Lynx Point USB 2.0 controller USB-A> mem 0xf7313000-0xf73133ff irq 23 at device 29.0 on pci0 usbus2: EHCI version 1.0 usbus2 on ehci1 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 ahci0: <Intel Lynx Point AHCI SATA controller> port 0xf050-0xf057,0xf040-0xf043,0xf030-0xf037,0xf020-0xf023,0xf000-0xf01f mem 0xf7312000-0xf73127ff irq 19 at device 31.2 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich3: <AHCI channel> at channel 3 on ahci0 ahcich4: <AHCI channel> at channel 4 on ahci0 ahcich5: <AHCI channel> at channel 5 on ahci0 acpi_button0: <Sleep Button> on acpi0 acpi_button1: <Power Button> on acpi0 acpi_tz0: <Thermal Zone> on acpi0 acpi_tz1: <Thermal Zone> on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart2: <16550 or compatible> port 0x3e8-0x3ef irq 7 on acpi0 ichwd0: <Intel Lynx Point watchdog timer> on isa0 ichwd0: ICH WDT present but disabled in BIOS or hardware device_attach: ichwd0 attach returned 6 wbwd0: <Nuvoton NCT6776 (0xc3/0x33) Watchdog Timer> at port 0x2e-0x2f on isa0 ichwd0: <Intel Lynx Point watchdog timer> at port 0x1830-0x1837,0x1860-0x187f on isa0 ichwd0: ICH WDT present but disabled in BIOS or hardware device_attach: ichwd0 attach returned 6 orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: CGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3d0-0x3db iomem 0xb8000-0xbffff on isa0 coretemp0: <CPU On-Die Thermal Sensors> on cpu0 est0: <Enhanced SpeedStep Frequency Control> on cpu0 coretemp1: <CPU On-Die Thermal Sensors> on cpu1 est1: <Enhanced SpeedStep Frequency Control> on cpu1 coretemp2: <CPU On-Die Thermal Sensors> on cpu2 est2: <Enhanced SpeedStep Frequency Control> on cpu2 coretemp3: <CPU On-Die Thermal Sensors> on cpu3 est3: <Enhanced SpeedStep Frequency Control> on cpu3 coretemp4: <CPU On-Die Thermal Sensors> on cpu4 est4: <Enhanced SpeedStep Frequency Control> on cpu4 coretemp5: <CPU On-Die Thermal Sensors> on cpu5 est5: <Enhanced SpeedStep Frequency Control> on cpu5 coretemp6: <CPU On-Die Thermal Sensors> on cpu6 est6: <Enhanced SpeedStep Frequency Control> on cpu6 coretemp7: <CPU On-Die Thermal Sensors> on cpu7 est7: <Enhanced SpeedStep Frequency Control> on cpu7 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled random: unblocking device. usbus0: 5.0Gbps Super Speed USB v3.0 usbus1: 480Mbps High Speed USB v2.0 usbus2: 480Mbps High Speed USB v2.0 ugen1.1: <Intel> at usbus1 uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 ugen0.1: <0x8086> at usbus0 uhub1: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 ugen2.1: <Intel> at usbus2 uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2 uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub1: 21 ports with 21 removable, self powered ugen1.2: <vendor 0x8087> at usbus1 uhub3: <vendor 0x8087 product 0x8008, class 9/0, rev 2.00/0.05, addr 2> on usbus1 ugen2.2: <vendor 0x8087> at usbus2 uhub4: <vendor 0x8087 product 0x8000, class 9/0, rev 2.00/0.05, addr 2> on usbus2 uhub3: 6 ports with 6 removable, self powered uhub4: 8 ports with 8 removable, self powered ugen0.2: <vendor 0x0000> at usbus0 uhub5: <vendor 0x0000 product 0x0001, class 9/0, rev 2.00/0.00, addr 1> on usbus0 uhub5: 4 ports with 3 removable, self powered ugen0.3: <vendor 0x0557> at usbus0 ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 2> on usbus0 kbd0 at ukbd0 ugen0.4: <SanDisk> at usbus0 umass0: <SanDisk Ultra Fit, class 0/0, rev 3.00/1.00, addr 3> on usbus0 umass0: SCSI over Bulk-Only; quirks = 0x8100 umass0:7:0:-1: Attached to scbus7 ugen0.5: <SanDisk> at usbus0 umass1: <SanDisk Ultra Fit, class 0/0, rev 3.00/1.00, addr 4> on usbus0 umass1: SCSI over Bulk-Only; quirks = 0x8100 umass1:8:1:-1: Attached to scbus8 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: <WDC WD30EFRX-68EUZN0 80.00A80> ACS-2 ATA SATA 3.x device ada0: Serial Number WD-WMC4N1351504 ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 2861588MB (5860533168 512 byte sectors) ada0: quirks=0x1<4K> ada0: Previously was known as ad4 ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: <WDC WD30EFRX-68EUZN0 80.00A80> ACS-2 ATA SATA 3.x device ada1: Serial Number WD-WMC4N1351789 ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 2861588MB (5860533168 512 byte sectors) ada1: quirks=0x1<4K> ada1: Previously was known as ad6 ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 ada2: <WDC WD30EFRX-68AX9N0 80.00A80> ACS-2 ATA SATA 3.x device ada2: Serial Number WD-WMC1T0928803 ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 2861588MB (5860533168 512 byte sectors) ada2: quirks=0x1<4K> ada2: Previously was known as ad8 ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 ada3: <WDC WD30EFRX-68AX9N0 80.00A80> ACS-2 ATA SATA 3.x device ada3: Serial Number WD-WMC1T1969409 ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 2861588MB (5860533168 512 byte sectors) ada3: quirks=0x1<4K> ada3: Previously was known as ad10 ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 ada4: <WDC WD30EFRX-68EUZN0 82.00A82> ACS-2 ATA SATA 3.x device ada4: Serial Number WD-WCC4N3VR2FC5 ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada4: Command Queueing enabled ada4: 2861588MB (5860533168 512 byte sectors) ada4: quirks=0x1<4K> ada4: Previously was known as ad12 ada5 at ahcich5 bus 0 scbus5 target 0 lun 0 ada5: <WDC WD30EFRX-68EUZN0 82.00A82> ACS-2 ATA SATA 3.x device ada5: Serial Number WD-WCC4N1SSDN1L ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada5: Command Queueing enabled ada5: 2861588MB (5860533168 512 byte sectors) ada5: quirks=0x1<4K> ada5: Previously was known as ad14 da0 at umass-sim0 bus 0 scbus7 target 0 lun 0 da0: <SanDisk Ultra Fit 1.00> Removable Direct Access SPC-4 SCSI device da0: Serial Number 4C531001440312110550 da0: 400.000MB/s transfers da0: 14832MB (30375936 512 byte sectors) da0: quirks=0x2<NO_6_BYTE> da1 at umass-sim1 bus 1 scbus8 target 0 lun 0 da1: <SanDisk Ultra Fit 1.00> Removable Direct Access SPC-4 SCSI device da1: Serial Number 4C530001050313115060 da1: 400.000MB/s transfers da1: 14832MB (30375936 512 byte sectors) da1: quirks=0x2<NO_6_BYTE> SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #7 Launched! Timecounter "TSC-low" frequency 1700036926 Hz quality 1000 Trying to mount root from zfs:freenas-boot/ROOT/FreeNAS-0a90dd7369b81ac71925bc9d69cb57e0 []... GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7) igb0: link state changed to UP ipmi0: <IPMI System Interface> port 0xca2,0xca3 on acpi0 ipmi0: KCS mode found at io 0xca2 on acpi ipmi0: IPMI device rev. 1, firmware rev. 1.42, version 2.0 ipmi0: Number of channels 2 ipmi0: Attached watchdog ichwd0: <Intel Lynx Point watchdog timer> at port 0x1830-0x1837,0x1860-0x187f on isa0 ichwd0: ICH WDT present but disabled in BIOS or hardware device_attach: ichwd0 attach returned 6 hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI> UCP/8/48/0x3f8<EDG,THR,REA,WRI,INV,QUA,PRC> UCF/1/48/0x60<REA,WRI> igb0: link state changed to DOWN igb0: link state changed to UP ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 2> on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 vboxdrv: fAsync=0 offMin=0x35e offMax=0xf1e GEOM_ELI: Device ada0p1.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware GEOM_ELI: Device ada1p1.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware GEOM_ELI: Device ada2p1.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware GEOM_ELI: Device ada3p1.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware GEOM_ELI: Device ada4p1.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware GEOM_ELI: Device ada5p1.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware nfsd: can't register svc name
"smartctl --scan" doesn't list the disks:
Code:
/dev/ada0 -d atacam # /dev/ada0, ATA device /dev/ada1 -d atacam # /dev/ada1, ATA device /dev/ada2 -d atacam # /dev/ada2, ATA device /dev/ada3 -d atacam # /dev/ada3, ATA device /dev/ada4 -d atacam # /dev/ada4, ATA device /dev/ada5 -d atacam # /dev/ada5, ATA device
Last edited: