Unable to mount pool

[judgereinhold]

Hello,

I've recently come into trouble after a power outage with my FreeNAS 8 instance.

My volume, named "vault", will not mount on boot.

I've tried exporting and re-importing the volume to no success.

Logged into the command line, I've issued the following:

Code:

# zpool import
  pool: vault
    id: 4788133954803953336
 state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

        vault       ONLINE
          raidz1    ONLINE
            ada0p2  ONLINE
            ada3p2  ONLINE
            ada2p2  ONLINE
            ada1p2  ONLINE
            ada4p2  ONLINE

So the drives appear to be healthy...

However:

Code:

# zpool import vault
cannot import 'vault': one or more devices is currently unavailable
# zpool import -f vault
cannot import 'vault': one or more devices is currently unavailable
# zpool import -R /mnt vault
cannot import 'vault': one or more devices is currently unavailable

And the problem exists on the GUI as well...

Doing an autoimport results in a [MiddlewareError: The volume "vault" failed to import, for further detail check pool status"]

Are there any kind of diagnostics I can run to determine what the problem is?

 

[peterh]

Are all expected drives there and functioning ?
the command "camcontrol devlist" might be informative, also "dmesg | less" will describe the systems view of hardware.

 

[judgereinhold]

Thanks for the reply.

As far as I can tell, the drives seem healthy, but unable to mount.

Code:

# camcontrol devlist
<Hitachi HDS722020ALA330 JKAOA28A>  at scbus0 target 0 lun 0 (pass0,ada0)
<Hitachi HDS722020ALA330 JKAOA3MA>  at scbus1 target 0 lun 0 (pass1,ada1)
<Hitachi HDS722020ALA330 JKAOA3EA>  at scbus2 target 0 lun 0 (pass2,ada2)
<ST32000540AS CC83>                at scbus3 target 0 lun 0 (pass3,ada3)
<ST32000644NS SN12>                at scbus4 target 0 lun 0 (pass4,ada4)
<WDC WD1600JB-00GVA0 08.02D08>     at scbus6 target 0 lun 0 (pass5,ada5)

and dmesg:

Code:

# dmesg | less
Copyright (c) 1992-2011 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.2-RELEASE-p3 #7: Fri Sep 30 12:51:49 PDT 2011
    jpaetzel@servant.iXsystems.com:/b/sf_freenas_build/obj.amd64/b/sf_freenas_build/FreeBSD/src/sys/FREENAS.amd64 amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Phenom(tm) II X4 955 Processor (3214.23-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x100f43  Family = 10  Model = 4  Stepping = 3
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802009<SSE3,MON,CX16,POPCNT>
  AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT>
  TSC: P-state invariant
real memory  = 8589934592 (8192 MB)
avail memory = 7708913664 (7351 MB)
ACPI APIC Table: <GBT    GBTUACPI>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
netsmb_dev: loaded
cryptosoft0: <software crypto> on motherboard
acpi0: <GBT GBTUACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, afce0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xee00-0xeeff mem 0xd0000000-0xdfffffff,0xfdee0000-0xfdeeffff,0xfdd00000-0xfddfffff irq 18 at device 5.0 on pci1
pci1: <multimedia, HDA> at device 5.1 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 5.0 on pci0
pci2: <ACPI PCI bus> on pcib2
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xfdcfe000-0xfdcfffff irq 17 at device 0.0 on pci2
xhci0: [ITHREAD]
usbus0 on xhci0
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 10.0 on pci0
pci3: <ACPI PCI bus> on pcib3
re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port 0xce00-0xceff mem 0xfdfff000-0xfdffffff,0xfdff8000-0xfdffbfff irq 18 at device 0.0 on pci3
re0: Using 1 MSI-X message
re0: Chip rev. 0x28000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0:  10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 6c:f0:49:72:d8:dd
re0: [ITHREAD]
ahci0: <ATI IXP700 AHCI SATA controller> port 0xff00-0xff07,0xfe00-0xfe03,0xfd00-0xfd07,0xfc00-0xfc03,0xfb00-0xfb0f mem 0xfe02f000-0xfe02f3ff irq 22 at device 17.0 on pci0
ahci0: [ITHREAD]
ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich0: [ITHREAD]
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich1: [ITHREAD]
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich2: [ITHREAD]
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich3: [ITHREAD]
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich4: [ITHREAD]
ahcich5: <AHCI channel> at channel 5 on ahci0
ahcich5: [ITHREAD]
ohci0: <OHCI (generic) USB controller> mem 0xfe02e000-0xfe02efff irq 16 at device 18.0 on pci0
ohci0: [ITHREAD]
usbus1: <OHCI (generic) USB controller> on ohci0
ohci1: <OHCI (generic) USB controller> mem 0xfe02d000-0xfe02dfff irq 16 at device 18.1 on pci0
ohci1: [ITHREAD]
usbus2: <OHCI (generic) USB controller> on ohci1
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfe02c000-0xfe02c0ff irq 17 at device 18.2 on pci0
ehci0: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <EHCI (generic) USB 2.0 controller> on ehci0
ohci2: <OHCI (generic) USB controller> mem 0xfe02b000-0xfe02bfff irq 18 at device 19.0 on pci0
ohci2: [ITHREAD]
usbus4: <OHCI (generic) USB controller> on ohci2
ohci3: <OHCI (generic) USB controller> mem 0xfe02a000-0xfe02afff irq 18 at device 19.1 on pci0
ohci3: [ITHREAD]
usbus5: <OHCI (generic) USB controller> on ohci3
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfe029000-0xfe0290ff irq 19 at device 19.2 on pci0
ehci1: [ITHREAD]
usbus6: EHCI version 1.0
usbus6: <EHCI (generic) USB 2.0 controller> on ehci1
pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
atapci0: <ATI IXP700/800 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfa00-0xfa0f at device 20.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
pci0: <multimedia, HDA> at device 20.2 (no driver attached)
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib4: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci4: <ACPI PCI bus> on pcib4
fwohci0: <Texas Instruments TSB43AB23> mem 0xfdaff000-0xfdaff7ff,0xfdaf8000-0xfdafbfff irq 22 at device 14.0 on pci4
fwohci0: [ITHREAD]
fwohci0: OHCI version 1.10 (ROM=0)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:36:09:a5:00:6c:f0:49
fwohci0: Phy 1394a available S400, 3 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
sbp0: <SBP-2/SCSI over FireWire> on firewire0
fwohci0: Initiate bus reset
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode
ohci4: <OHCI (generic) USB controller> mem 0xfe028000-0xfe028fff irq 18 at devic
e 20.5 on pci0
ohci4: [ITHREAD]
usbus7: <OHCI (generic) USB controller> on ohci4
amdtemp0: <AMD K8 Thermal Sensors> on hostb4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounters tick every 1.000 msec
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0)  (me)
firewire0: bus manager 0
usbus0: 4.8Gbps Super Speed USB v3.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 480Mbps High Speed USB v2.0
usbus7: 12Mbps Full Speed USB v1.0
ugen0.1: <0x1033> at usbus0
uhub0: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ugen1.1: <ATI> at usbus1
uhub1: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <ATI> at usbus2
uhub2: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <ATI> at usbus3
uhub3: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <ATI> at usbus4
uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <ATI> at usbus5
uhub5: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <ATI> at usbus6
uhub6: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6
ugen7.1: <ATI> at usbus7
uhub7: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus7
uhub7: 2 ports with 2 removable, self powered
uhub1: 3 ports with 3 removable, self powered
uhub2: 3 ports with 3 removable, self powered
uhub4: 3 ports with 3 removable, self powered
uhub5: 3 ports with 3 removable, self powered
uhub0: 4 ports with 4 removable, self powered
uhub3: 6 ports with 6 removable, self powered
uhub6: 6 ports with 6 removable, self powered
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <Hitachi HDS722020ALA330 JKAOA3MA> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <Hitachi HDS722020ALA330 JKAOA3EA> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <ST32000540AS CC83> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada4: <ST32000644NS SN12> ATA-8 SATA 2.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5 at ata0 bus 0 scbus6 target 0 lun 0
ada5: <WDC WD1600JB-00GVA0 08.02D08> ATA-6 device
ada5: 100.000MB/s transfers (UDMA5, PIO 8192bytes)
ada5: 152626MB (312579695 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Trying to mount root from ufs:/dev/ufs/FreeNASs1a

The most relevant part being

Code:

ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <Hitachi HDS722020ALA330 JKAOA3MA> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <Hitachi HDS722020ALA330 JKAOA3EA> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <ST32000540AS CC83> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada4: <ST32000644NS SN12> ATA-8 SATA 2.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5 at ata0 bus 0 scbus6 target 0 lun 0
ada5: <WDC WD1600JB-00GVA0 08.02D08> ATA-6 device
ada5: 100.000MB/s transfers (UDMA5, PIO 8192bytes)
ada5: 152626MB (312579695 512 byte sectors: 16H 63S/T 16383C)

So it seems like they're being initialized.

Is there any kind of cache or DB I need to rebuild? I've searched for instances of my pool "vault" in /data and got a few hits on a couple binary files, but I'm too concerned with losing data to muck around with them without knowing what I'm doing.

 

[peterh]

one way might be to use another instance of freenas/freebsd/solaris and try to import to that instance.

As one instance you could try to copy your freebsd stick, boot from the new one and "reset to factory default" and then import the disks. Then there only depend on the disks themself, not from any configuration
database.

 

[judgereinhold]

one way might be to use another instance of freenas/freebsd/solaris and try to import to that instance.

As one instance you could try to copy your freebsd stick, boot from the new one and "reset to factory default" and then import the disks. Then there only depend on the disks themself, not from any configuration
database.

Thanks for the reply. I've rebuilt the OS and tried reimporting and same results...

Are there any suggestions, or am I SOL?

Any guides or recommendations for migrating the data off the disks?

 

[peterh]

I'm out of ideas. Time to roll back the backup.

 

[campbela]

Thanks for the reply. I've rebuilt the OS and tried reimporting and same results...

Did you try all of the suggestions or just rebuilding FreeNAS?

one way might be to use another instance of freenas/freebsd/solaris and try to import to that instance.

If not, then I would try this:

http://www.pcbsd.org/

 

[William Grzybowski]

Yeah, I would definitely try pcbsd 9, or solaris or linux (zfs on linux)

 

[judgereinhold]

Yeah, I would definitely try pcbsd 9, or solaris or linux (zfs on linux)

I installed Ubuntu over the disk that was the FreeNAS OS, and installed a zfs RPM. The import gave a little bit more info...

"vault pool may be in use from other system, it was last accessed by (...) on Jan 15, use -f to import anyway"

Unfortunately doing a -f on the import does not appear to work, 100% cpu, let it go for about 12 hours, no end.

I'm thinking of seeing if system rescue cd has any effect, but I'm afraid it's looking bleak.

 

[campbela]

There is some good information in this thread:

http://forums.freenas.org/showthread.php?395-zpool-import-hangs-forever-help-please!!

 

[judgereinhold]

Thanks all.

It worked after installing PC BSD, then running the
zpool import -fFX vault

and doing a scrub.

Then it imported in FreNAS after being repaired.

Thank you everyone!