Kernel panics when accessing one directory despite successful scrub

Status
Not open for further replies.
A

attax

Cadet
Joined
Jan 19, 2014
Messages
2
Hi all!

My HP N36L system with 8GB RAM and 4x 3TB drives in RAIDZ1 config has suddenly become unstable after 2.5 years. My main volume "bigmem" contains, among others, a directory named "eyeTV". Whenever I access files in this directory, by network or locally in the shell, I get an immediate Kernel Panic and reboot. Below the log after rebooting.

Code:
Jan 19 20:11:17 freenas newsyslog[2095]: logfile first created
Jan 19 20:11:17 freenas syslogd: kernel boot file is /boot/kernel/kernel
Jan 19 20:11:17 freenas kernel: panic: Solaris(panic): zfs: link count on vnode 0xfffffe01f7409dc8 is 1, should be at least 2
Jan 19 20:11:17 freenas kernel: cpuid = 1
Jan 19 20:11:17 freenas kernel: KDB: enter: panic


This one above happened about 5 minutes after a successful scrub of the volume.
It seems like the referenced vnode changes from panic to panic, but it is reproducibly linked to accessing files in this directory "eyeTV". I tried "rm -r *" within "eyeTV" from the local console, but this led to Kernel Panic, as well.

I am willing to sacrifice the "eyeTV" directory and everything in it, but I would really like to keep the rest of the data on "bigmem".

The way I've understood it, a scrub is similar to a fsck in that it validates the integrity of the file system. I am not sure if a successful scrub should even be able to leave me with an unstable file system.

Note that I upgraded to FreeNAS 9.2 from 8.1 recently, and I just added my first plugin (ownCloud) in a jail previous to the issues arising. IMHO that is not related, but what do I know...

Also, one of the 4 drives has been having unrecoverable sectors, but these were corrected in the scrub with no data loss. I fear that, right now, replacing that drive and resilvering, would either a) not help, or even b) fail catastrophically when it hits the directory, leading to a bigger problem.
 
C

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Do you get a kernel panic if you go to the eyeTV directory from the CLI?

Post the output of:

dmesg
zpool status
 
A

attax

Cadet
Joined
Jan 19, 2014
Messages
2
Hello cyberjock:

good question - you pinned down exactly where I was unclear. I think I've narrowed it down to these two items in the file system, where "ls" and "ls -la" give the following output (and I deleted everything else in "eyeTV" along the way).

Code:
[root@freenas /mnt/bigmem/user/eyeTV/EyeTV Archive/Live TV Buffer.eyetv]# ls                                                       
.AppleDouble            0000000014fc37e3.eyetvi                                                                                   
[root@freenas /mnt/bigmem/user/eyeTV/EyeTV Archive/Live TV Buffer.eyetv]# ls -la                                                   
ls: 0000000014fc37e3.eyetvi: No such file or directory                                                                             
total 166                                                                                                                         
drwx--S---  3 axel  family  4 Jan 18 11:47 .                                                                                       
drwx--S---  3 axel  family  3 Jan 19 22:24 ..                                                                                     
drwx--S---  1 axel  family  2 Jan 18 11:47 .AppleDouble                                                                           
[root@freenas /mnt/bigmem/user/eyeTV/EyeTV Archive/Live TV Buffer.eyetv]#


Trying to "rm -rf .A*" to remove the ".AppleDouble" gave me a Kernel Panic.
I don't know what to do about the "oooo...eyetvi" file/thing, either.

So far, FreeNAS seems to cope quite well with these KPs/reboots, but I don't want to overdo it. I do hope it is a logical error in the file system that can be fixed, not something (actively) corrupting behind the scenes...

Other requested outputs included below. Any comments appreciated!

Code:
[root@freenas ~]# zpool status                                                                                                     
  pool: bigmem                                                                                                                     
state: ONLINE                                                                                                                     
status: The pool is formatted using a legacy on-disk format.  The pool can                                                         
        still be used, but some features are unavailable.                                                                         
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the                                                           
        pool will no longer be accessible on software that does not support feature                                               
        flags.                                                                                                                     
  scan: scrub repaired 728K in 20h4m with 0 errors on Sun Jan 19 20:02:15 2014                                                     
config:                                                                                                                           
                                                                                                                                   
        NAME                                            STATE    READ WRITE CKSUM                                                 
        bigmem                                          ONLINE      0    0    0                                                 
          raidz1-0                                      ONLINE      0    0    0                                                 
            gptid/f38872bd-fcad-11e0-a268-3cd92b029763  ONLINE      0    0    0                                                 
            gptid/f49b0678-fcad-11e0-a268-3cd92b029763  ONLINE      0    0    0                                                 
            gptid/f5b8a02e-fcad-11e0-a268-3cd92b029763  ONLINE      0    0    0                                                 
            gptid/f6d1e13a-fcad-11e0-a268-3cd92b029763  ONLINE      0    0    0                                                 
                                                                                                                                   
errors: No known data errors


Code:
Jan 19 20:11:17 freenas newsyslog[2095]: logfile first created
Jan 19 20:11:17 freenas syslogd: kernel boot file is /boot/kernel/kernel
Jan 19 20:11:17 freenas kernel: panic: Solaris(panic): zfs: link count on vnode 0xfffffe01f7409dc8 is 1, should be at least 2
Jan 19 20:11:17 freenas kernel: cpuid = 1
Jan 19 20:11:17 freenas kernel: KDB: enter: panic
Jan 19 20:11:17 freenas kernel: Copyright (c) 1992-2013 The FreeBSD Project.
Jan 19 20:11:17 freenas kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jan 19 20:11:17 freenas kernel: The Regents of the University of California. All rights reserved.
Jan 19 20:11:17 freenas kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Jan 19 20:11:17 freenas kernel: FreeBSD 9.2-RELEASE #0 r+2315ea3: Fri Dec 20 12:48:50 PST 2013
Jan 19 20:11:17 freenas kernel: root@build.ixsystems.com:/tank/home/jkh/checkout/freenas/os-base/amd64/tank/home/jkh/checkout/freenas/FreeBSD/src/sys/FREENAS.amd64 amd64
Jan 19 20:11:17 freenas kernel: gcc version 4.2.1 20070831 patched [FreeBSD]
Jan 19 20:11:17 freenas kernel: CPU: AMD Athlon(tm) II Neo N36L Dual-Core Processor (1297.87-MHz K8-class CPU)
Jan 19 20:11:17 freenas kernel: Origin = "AuthenticAMD"  Id = 0x100f63  Family = 0x10  Model = 0x6  Stepping = 3
Jan 19 20:11:17 freenas kernel: Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Jan 19 20:11:17 freenas kernel: Features2=0x802009<SSE3,MON,CX16,POPCNT>
Jan 19 20:11:17 freenas kernel: AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
Jan 19 20:11:17 freenas kernel: AMD Features2=0x8377f<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,Prefetch,OSVW,IBS,SKINIT,WDT,NodeId>
Jan 19 20:11:17 freenas kernel: TSC: P-state invariant
Jan 19 20:11:17 freenas kernel: real memory  = 8589934592 (8192 MB)
Jan 19 20:11:17 freenas kernel: avail memory = 8110198784 (7734 MB)
Jan 19 20:11:17 freenas kernel: Event timer "LAPIC" quality 400
Jan 19 20:11:17 freenas kernel: ACPI APIC Table: <HP    ProLiant>
Jan 19 20:11:17 freenas kernel: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
Jan 19 20:11:17 freenas kernel: FreeBSD/SMP: 1 package(s) x 2 core(s)
Jan 19 20:11:17 freenas kernel: cpu0 (BSP): APIC ID:  0
Jan 19 20:11:17 freenas kernel: cpu1 (AP): APIC ID:  1
Jan 19 20:11:17 freenas kernel: WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
Jan 19 20:11:17 freenas kernel: ioapic0 <Version 2.1> irqs 0-23 on motherboard
Jan 19 20:11:17 freenas kernel: kbd1 at kbdmux0
Jan 19 20:11:17 freenas kernel: cryptosoft0: <software crypto> on motherboard
Jan 19 20:11:17 freenas kernel: aesni0: No AESNI support.
Jan 19 20:11:17 freenas kernel: padlock0: No ACE support.
Jan 19 20:11:17 freenas kernel: acpi0: <HP ProLiant> on motherboard
Jan 19 20:11:17 freenas kernel: acpi0: Power Button (fixed)
Jan 19 20:11:17 freenas kernel: acpi0: reservation of fee00000, 1000 (3) failed
Jan 19 20:11:17 freenas kernel: acpi0: reservation of ffb80000, 80000 (3) failed
Jan 19 20:11:17 freenas kernel: acpi0: reservation of fec10000, 20 (3) failed
Jan 19 20:11:17 freenas kernel: acpi0: reservation of fed80000, 1000 (3) failed
Jan 19 20:11:17 freenas kernel: acpi0: reservation of 0, a0000 (3) failed
Jan 19 20:11:17 freenas kernel: acpi0: reservation of 100000, d7f00000 (3) failed
Jan 19 20:11:17 freenas kernel: cpu0: <ACPI CPU> on acpi0
Jan 19 20:11:17 freenas kernel: cpu1: <ACPI CPU> on acpi0
Jan 19 20:11:17 freenas kernel: attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Jan 19 20:11:17 freenas kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Jan 19 20:11:17 freenas kernel: Event timer "i8254" frequency 1193182 Hz quality 100
Jan 19 20:11:17 freenas kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Jan 19 20:11:17 freenas kernel: Event timer "RTC" frequency 32768 Hz quality 0
Jan 19 20:11:17 freenas kernel: hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Jan 19 20:11:17 freenas kernel: Timecounter "HPET" frequency 14318180 Hz quality 950
Jan 19 20:11:17 freenas kernel: Event timer "HPET" frequency 14318180 Hz quality 550
Jan 19 20:11:17 freenas kernel: Event timer "HPET1" frequency 14318180 Hz quality 450
Jan 19 20:11:17 freenas kernel: Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
Jan 19 20:11:17 freenas kernel: acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
Jan 19 20:11:17 freenas kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Jan 19 20:11:17 freenas kernel: pci0: <ACPI PCI bus> on pcib0
Jan 19 20:11:17 freenas kernel: pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
Jan 19 20:11:17 freenas kernel: pci1: <ACPI PCI bus> on pcib1
Jan 19 20:11:17 freenas kernel: vgapci0: <VGA-compatible display> port 0xd000-0xd0ff mem 0xf0000000-0xf7ffffff,0xfe7f0000-0xfe7fffff,0xfe600000-0xfe6fffff irq 18 at device 5.0 on pci1
Jan 19 20:11:17 freenas kernel: pcib2: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
Jan 19 20:11:17 freenas kernel: pci2: <ACPI PCI bus> on pcib2
Jan 19 20:11:17 freenas kernel: em0: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0xe800-0xe81f mem 0xfe8e0000-0xfe8fffff,0xfe800000-0xfe87ffff,0xfe8dc000-0xfe8dffff irq 16 at device 0.0 on pci2
Jan 19 20:11:17 freenas kernel: em0: Using MSIX interrupts with 3 vectors
Jan 19 20:11:17 freenas kernel: em0: Ethernet address: 68:05:ca:00:e7:41
Jan 19 20:11:17 freenas kernel: pcib3: <ACPI PCI-PCI bridge> irq 18 at device 6.0 on pci0
Jan 19 20:11:17 freenas kernel: pci3: <ACPI PCI bus> on pcib3
Jan 19 20:11:17 freenas kernel: bge0: <HP NC107i PCIe Gigabit Server Adapter, ASIC rev. 0x5784100> mem 0xfe9f0000-0xfe9fffff irq 18 at device 0.0 on pci3
Jan 19 20:11:17 freenas kernel: bge0: CHIP ID 0x05784100; ASIC REV 0x5784; CHIP REV 0x57841; PCI-E
Jan 19 20:11:17 freenas kernel: miibus0: <MII bus> on bge0
Jan 19 20:11:17 freenas kernel: brgphy0: <BCM5784 10/100/1000baseT PHY> PHY 1 on miibus0
Jan 19 20:11:17 freenas kernel: brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
Jan 19 20:11:17 freenas kernel: bge0: Ethernet address: 3c:d9:2b:02:97:63
Jan 19 20:11:17 freenas kernel: ahci0: <ATI IXP700 AHCI SATA controller> port 0xc000-0xc007,0xb000-0xb003,0xa000-0xa007,0x9000-0x9003,0x8000-0x800f mem 0xfe5ffc00-0xfe5fffff irq 19 at device 17.0 on pci0
Jan 19 20:11:17 freenas kernel: ahci0: AHCI v1.20 with 4 3Gbps ports, Port Multiplier supported
Jan 19 20:11:17 freenas kernel: ahcich0: <AHCI channel> at channel 0 on ahci0
Jan 19 20:11:17 freenas kernel: ahcich1: <AHCI channel> at channel 1 on ahci0
Jan 19 20:11:17 freenas kernel: ahcich2: <AHCI channel> at channel 2 on ahci0
Jan 19 20:11:17 freenas kernel: ahcich3: <AHCI channel> at channel 3 on ahci0
Jan 19 20:11:17 freenas kernel: ohci0: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe5fe000-0xfe5fefff irq 18 at device 18.0 on pci0
Jan 19 20:11:17 freenas kernel: usbus0 on ohci0
Jan 19 20:11:17 freenas kernel: ehci0: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfe5ff800-0xfe5ff8ff irq 17 at device 18.2 on pci0
Jan 19 20:11:17 freenas kernel: usbus1: EHCI version 1.0
Jan 19 20:11:17 freenas kernel: usbus1 on ehci0
Jan 19 20:11:17 freenas kernel: ohci1: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe5fd000-0xfe5fdfff irq 18 at device 19.0 on pci0
Jan 19 20:11:17 freenas kernel: usbus2 on ohci1
Jan 19 20:11:17 freenas kernel: ehci1: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfe5ff400-0xfe5ff4ff irq 17 at device 19.2 on pci0
Jan 19 20:11:17 freenas kernel: usbus3: EHCI version 1.0
Jan 19 20:11:17 freenas kernel: usbus3 on ehci1
Jan 19 20:11:17 freenas kernel: pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
Jan 19 20:11:17 freenas kernel: atapci0: <ATI IXP700/800 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0
Jan 19 20:11:17 freenas kernel: ata0: <ATA channel> at channel 0 on atapci0
Jan 19 20:11:17 freenas kernel: ata1: <ATA channel> at channel 1 on atapci0
Jan 19 20:11:17 freenas kernel: isab0: <PCI-ISA bridge> at device 20.3 on pci0
Jan 19 20:11:17 freenas kernel: isa0: <ISA bus> on isab0
Jan 19 20:11:17 freenas kernel: pcib4: <ACPI PCI-PCI bridge> at device 20.4 on pci0
Jan 19 20:11:17 freenas kernel: pci4: <ACPI PCI bus> on pcib4
Jan 19 20:11:17 freenas kernel: ohci2: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe5fc000-0xfe5fcfff irq 18 at device 22.0 on pci0
Jan 19 20:11:17 freenas kernel: usbus4 on ohci2
Jan 19 20:11:17 freenas kernel: ehci2: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfe5ff000-0xfe5ff0ff irq 17 at device 22.2 on pci0
Jan 19 20:11:17 freenas kernel: usbus5: EHCI version 1.0
Jan 19 20:11:17 freenas kernel: usbus5 on ehci2
Jan 19 20:11:17 freenas kernel: amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb4
Jan 19 20:11:17 freenas kernel: acpi_button0: <Power Button> on acpi0
Jan 19 20:11:17 freenas kernel: orm0: <ISA Option ROM> at iomem 0xd2000-0xd2fff on isa0
Jan 19 20:11:17 freenas kernel: amdsbwd0: <AMD SB8xx Watchdog Timer> at iomem 0xfec000f0-0xfec000f3,0xfec000f4-0xfec000f7 on isa0
Jan 19 20:11:17 freenas kernel: sc0: <System console> at flags 0x100 on isa0
Jan 19 20:11:17 freenas kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Jan 19 20:11:17 freenas kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Jan 19 20:11:17 freenas kernel: ppc0: cannot reserve I/O port range
Jan 19 20:11:17 freenas kernel: wbwd0: HEFRAS and EFER do not align: EFER 0x2e DevID 0xff DevRev 0xff CR26 0xff
Jan 19 20:11:17 freenas kernel: acpi_throttle0: <ACPI CPU Throttling> on cpu0
Jan 19 20:11:17 freenas kernel: hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Jan 19 20:11:17 freenas kernel: Timecounters tick every 1.000 msec
Jan 19 20:11:17 freenas kernel: ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
Jan 19 20:11:17 freenas kernel: DUMMYNET 0xfffffe0002f52300 with IPv6 initialized (100409)
Jan 19 20:11:17 freenas kernel: load_dn_sched dn_sched QFQ loaded
Jan 19 20:11:17 freenas kernel: load_dn_sched dn_sched RR loaded
Jan 19 20:11:17 freenas kernel: load_dn_sched dn_sched WF2Q+ loaded
Jan 19 20:11:17 freenas kernel: load_dn_sched dn_sched FIFO loaded
Jan 19 20:11:17 freenas kernel: load_dn_sched dn_sched PRIO loaded
Jan 19 20:11:17 freenas kernel: usbus0: 12Mbps Full Speed USB v1.0
Jan 19 20:11:17 freenas kernel: usbus1: 480Mbps High Speed USB v2.0
Jan 19 20:11:17 freenas kernel: usbus2: 12Mbps Full Speed USB v1.0
Jan 19 20:11:17 freenas kernel: usbus3: 480Mbps High Speed USB v2.0
Jan 19 20:11:17 freenas kernel: usbus4: 12Mbps Full Speed USB v1.0
Jan 19 20:11:17 freenas kernel: usbus5: 480Mbps High Speed USB v2.0
Jan 19 20:11:17 freenas kernel: ugen0.1: <ATI> at usbus0
Jan 19 20:11:17 freenas kernel: uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
Jan 19 20:11:17 freenas kernel: ugen5.1: <ATI> at usbus5
Jan 19 20:11:17 freenas kernel: uhub1: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
Jan 19 20:11:17 freenas kernel: ugen4.1: <ATI> at usbus4
Jan 19 20:11:17 freenas kernel: uhub2: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
Jan 19 20:11:17 freenas kernel: ugen3.1: <ATI> at usbus3
Jan 19 20:11:17 freenas kernel: uhub3: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
Jan 19 20:11:17 freenas kernel: ugen2.1: <ATI> at usbus2
Jan 19 20:11:17 freenas kernel: uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
Jan 19 20:11:17 freenas kernel: ugen1.1: <ATI> at usbus1
Jan 19 20:11:17 freenas kernel: uhub5: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
Jan 19 20:11:17 freenas kernel: uhub2: 4 ports with 4 removable, self powered
Jan 19 20:11:17 freenas kernel: uhub0: 5 ports with 5 removable, self powered
Jan 19 20:11:17 freenas kernel: uhub4: 5 ports with 5 removable, self powered
Jan 19 20:11:17 freenas kernel: uhub1: 4 ports with 4 removable, self powered
Jan 19 20:11:17 freenas kernel: uhub3: 5 ports with 5 removable, self powered
Jan 19 20:11:17 freenas kernel: uhub5: 5 ports with 5 removable, self powered
Jan 19 20:11:17 freenas kernel: ugen2.2: <CHESEN> at usbus2
Jan 19 20:11:17 freenas kernel: ukbd0: <CHESEN PS2 to USB Converter, class 0/0, rev 1.10/0.10, addr 2> on usbus2
Jan 19 20:11:17 freenas kernel: kbd0 at ukbd0
Jan 19 20:11:17 freenas kernel: ums0: <CHESEN PS2 to USB Converter, class 0/0, rev 1.10/0.10, addr 2> on usbus2
Jan 19 20:11:17 freenas kernel: ums0: 5 buttons and [XYZ] coordinates ID=1
Jan 19 20:11:17 freenas kernel: ugen3.2: <Verbatim> at usbus3
Jan 19 20:11:17 freenas kernel: umass0: <Verbatim STORE N GO, class 0/0, rev 2.00/1.00, addr 2> on usbus3
Jan 19 20:11:17 freenas kernel: ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
Jan 19 20:11:17 freenas kernel: ada0: <WDC WD30EZRX-00MMMB0 80.00A80> ATA-8 SATA 3.x device
Jan 19 20:11:17 freenas kernel: ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Jan 19 20:11:17 freenas kernel: ada0: Command Queueing enabled
Jan 19 20:11:17 freenas kernel: ada0: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
Jan 19 20:11:17 freenas kernel: ada0: quirks=0x1<4K>
Jan 19 20:11:17 freenas kernel: ada0: Previously was known as ad4
Jan 19 20:11:17 freenas kernel: ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
Jan 19 20:11:17 freenas kernel: ada1: <WDC WD30EZRX-00MMMB0 80.00A80> ATA-8 SATA 3.x device
Jan 19 20:11:17 freenas kernel: ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Jan 19 20:11:17 freenas kernel: ada1: Command Queueing enabled
Jan 19 20:11:17 freenas kernel: ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
Jan 19 20:11:17 freenas kernel: ada1: quirks=0x1<4K>
Jan 19 20:11:17 freenas kernel: ada1: Previously was known as ad6
Jan 19 20:11:17 freenas kernel: ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
Jan 19 20:11:17 freenas kernel: ada2: <WDC WD30EZRX-00MMMB0 80.00A80> ATA-8 SATA 3.x device
Jan 19 20:11:17 freenas kernel: ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Jan 19 20:11:17 freenas kernel: ada2: Command Queueing enabled
Jan 19 20:11:17 freenas kernel: ada2: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
Jan 19 20:11:17 freenas kernel: ada2: quirks=0x1<4K>
Jan 19 20:11:17 freenas kernel: ada2: Previously was known as ad8
Jan 19 20:11:17 freenas kernel: ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
Jan 19 20:11:17 freenas kernel: ada3: <WDC WD30EZRX-00MMMB0 80.00A80> ATA-8 SATA 3.x device
Jan 19 20:11:17 freenas kernel: ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Jan 19 20:11:17 freenas kernel: ada3: Command Queueing enabled
Jan 19 20:11:17 freenas kernel: ada3: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
Jan 19 20:11:17 freenas kernel: ada3: quirks=0x1<4K>
Jan 19 20:11:17 freenas kernel: ada3: Previously was known as ad10
Jan 19 20:11:17 freenas kernel: da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
Jan 19 20:11:17 freenas kernel: SMP: AP CPU #1 Launched!
Jan 19 20:11:17 freenas kernel: da0: <Verbatim STORE N GO 1.00> Removable Direct Access SCSI-2 device
Jan 19 20:11:17 freenas kernel: da0: 40.000MB/s transfers
Jan 19 20:11:17 freenas kernel: da0: 7645MB (15656960 512 byte sectors: 255H 63S/T 974C)
Jan 19 20:11:17 freenas kernel: da0: quirks=0x2<NO_6_BYTE>
Jan 19 20:11:17 freenas kernel: Timecounter "TSC" frequency 1297870679 Hz quality 800
Jan 19 20:11:17 freenas kernel: Trying to mount root from ufs:/dev/ufs/FreeNASs2a [ro]...
Jan 19 20:11:17 freenas kernel: WARNING: /data was not properly dismounted
Jan 19 20:11:17 freenas kernel: GEOM_RAID5: Module loaded, version 1.1.20130907.44 (rev 5c6d2a159411)
Jan 19 20:11:17 freenas kernel: ZFS filesystem version: 5
Jan 19 20:11:17 freenas kernel: ZFS storage pool version: features support (5000)
Jan 19 20:11:17 freenas kernel: GEOM_ELI: Device ada0p1.eli created.
Jan 19 20:11:17 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256
Jan 19 20:11:17 freenas kernel: GEOM_ELI:    Crypto: software
Jan 19 20:11:17 freenas kernel: GEOM_ELI: Device ada1p1.eli created.
Jan 19 20:11:17 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256
Jan 19 20:11:17 freenas kernel: GEOM_ELI:    Crypto: software
Jan 19 20:11:17 freenas kernel: GEOM_ELI: Device ada2p1.eli created.
Jan 19 20:11:17 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256
Jan 19 20:11:17 freenas kernel: GEOM_ELI:    Crypto: software
Jan 19 20:11:17 freenas kernel: GEOM_ELI: Device ada3p1.eli created.
Jan 19 20:11:17 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256
Jan 19 20:11:17 freenas kernel: GEOM_ELI:    Crypto: software
Jan 19 20:11:18 freenas root: /etc/rc: WARNING: failed precmd routine for vmware_guestd
Jan 19 20:11:18 freenas ntpd[2313]: ntpd 4.2.4p5-a (1)
Jan 19 20:11:19 freenas kernel: ..............................
Jan 19 20:11:19 freenas kernel: ............................+++
Jan 19 20:11:20 freenas kernel: .............................
Jan 19 20:11:20 freenas kernel: ..............................
Jan 19 20:11:20 freenas kernel: .............................
Jan 19 20:11:20 freenas kernel: ..............................
Jan 19 20:11:20 freenas kernel: .+++
Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada0, 2 Currently unreadable (pending) sectors
Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada0, 1 Offline uncorrectable sectors
Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada2, 1 Offline uncorrectable sectors
Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada3, 3 Currently unreadable (pending) sectors
Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada3, 21 Offline uncorrectable sectors
Jan 19 20:11:27 freenas ntpd[2314]: time reset -0.426175 s
Jan 19 20:11:29 freenas avahi-daemon[3338]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Jan 19 20:11:29 freenas netatalk[3342]: Netatalk AFP server starting
Jan 19 20:11:29 freenas cnid_metad[3363]: CNID Server listening on localhost:4700
Jan 19 20:11:31 freenas afpd[3362]: Netatalk AFP/TCP listening on 192.168.1.20:548
Jan 19 20:11:32 freenas kernel: 5
Jan 19 20:11:33 freenas kernel: 4
Jan 19 20:11:34 freenas kernel: done.
Jan 19 20:11:38 freenas kernel: bridge0: Ethernet address: 02:28:c3:a2:43:00
Jan 19 20:11:38 freenas kernel: bridge0: link state changed to UP
Jan 19 20:11:38 freenas kernel: em0: promiscuous mode enabled
Jan 19 20:11:38 freenas kernel: epair0a: Ethernet address: 02:65:f9:00:0c:0a
Jan 19 20:11:38 freenas kernel: epair0b: Ethernet address: 02:65:f9:00:0d:0b
Jan 19 20:11:38 freenas kernel: epair0a: link state changed to UP
Jan 19 20:11:38 freenas kernel: epair0b: link state changed to UP
Jan 19 20:11:38 freenas rtsold[1867]: <rtsock_input_ifannounce> interface epair0b removed
Jan 19 20:11:38 freenas kernel: em0: link state changed to DOWN
Jan 19 20:11:38 freenas kernel: epair0a: promiscuous mode enabled
Jan 19 20:11:41 freenas kernel: em0: link state changed to UP
Jan 19 20:14:07 freenas manage.py: [py.warnings:193] /usr/local/lib/python2.7/site-packages/django/http/request.py:193: DeprecationWarning: HttpRequest.raw_post_data has been deprecated. Use HttpRequest.body instead.  warnings.warn('HttpRequest.raw_post_data has been deprecated. Use HttpRequest.body instead.', DeprecationWarning)


Note different vnode number after latest reboot:
Code:
Jan 19 22:18:08 freenas newsyslog[2095]: logfile first created
Jan 19 22:18:08 freenas syslogd: kernel boot file is /boot/kernel/kernel
Jan 19 22:18:08 freenas kernel: panic: Solaris(panic): zfs: link count on vnode 0xfffffe0086c54dc8 is 1, should be at least 2
Jan 19 22:18:08 freenas kernel: cpuid = 1
Jan 19 22:18:08 freenas kernel: KDB: enter: panic


Going through the pdf from your .sig now...
BR
attax
 
warri

warri

Guru
Joined
Jun 6, 2011
Messages
1,193
Two little comments (I'm sure cyberjock will repeat this):

1. Three of your hard drives have SMART errors:
Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada0, 2 Currently unreadable (pending) sectors
Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada0, 1 Offline uncorrectable sectors
Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada2, 1 Offline uncorrectable sectors
Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada3, 3 Currently unreadable (pending) sectors
Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada3, 21 Offline uncorrectable sectors
Click to expand...

Can you post the output of smartctl -a -q noserial adaX (substitute X with number) for each of those disks? Did you setup regular SMART tests?

2. The scrub repaired data, probably due to failing hard drives (see 1.).
 
C

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
So you have a problem...

  1. Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada0, 2 Currently unreadable (pending) sectors
  2. Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada0, 1 Offline uncorrectable sectors
  3. Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada2, 1 Offline uncorrectable sectors
  4. Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada3, 3 Currently unreadable (pending) sectors
  5. Jan 19 20:11:22 freenas smartd[2564]: Device: /dev/ada3, 21 Offline uncorrectable sectors
That shows that ada0, ada2, and ada3 have problems. That is quite possibly why you are having problems. You need to examine all 3 of those disks(I'm guessing RMAs for all 3 or replacement is necessary).

Sounds like you haven't properly setup all of the freenas features to monitor, test, and report disk problems.

At this point, I'd look at backing up all of your data and destroying the pool followed by disk diagnostics and replacements as necessary.
 
Status
Not open for further replies.

Similar threads

S
  • Locked
adding 3TB disc: middleware.exceptions:38 - Unable to GPT format the disk
Replies
10
Views
7K
stephenreda
S
J
  • Locked
Fatal Trap 12 when attempting to detach drive
Replies
3
Views
2K
jcooper
J
S
  • Locked
Resilvering horrible slow (takes weeks!!)
Replies
4
Views
3K
samweis
S
E
Kernel Panic During Scrub
Replies
0
Views
872
Exmo
E
Raymond Matos
  • Locked
Kernel panic when importing zfs pool
Replies
3
Views
1K
Robert Trevellyan
R
Top