ZFS scrub stuck, and won't continue

PCorsetti

Dabbler
Joined
Aug 28, 2019
Messages
19
We have a FreeNAS that go in stuck when we perform action on Pools.
My FreeNAS is the following:

OS Version:
FreeNAS-11.2-U4.1
(Build Date: May 10, 2019 21:33)
Processor:
Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz (32 cores)
Memory:
128 GiB
HostName:
freenas-slot08-e9000.cc.local
Uptime:
11:59AM up 2:09, 2 users


Investigating I have found that the Scrub task is locked at 9.72% and doesn't continue.

This is the result of zpool status command:


root@freenas-slot08-e9000:~ # zpool status vol1_raidz2_freenas2
pool: vol1_raidz2_freenas2
state: ONLINE
scan: scrub in progress since Sun Jul 21 00:00:05 2019
1.36T scanned at 0/s, 1.36T issued at 0/s, 14.0T total
0 repaired, 9.72% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
vol1_raidz2_freenas2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/926a799e-67e4-11e7-9711-e03676db32c5 ONLINE 0 0 0
gptid/93151f52-67e4-11e7-9711-e03676db32c5 ONLINE 0 0 0
gptid/93c2328a-67e4-11e7-9711-e03676db32c5 ONLINE 0 0 0
gptid/9484d9d0-67e4-11e7-9711-e03676db32c5 ONLINE 0 0 0
gptid/e76eeb0c-e1c3-11e7-b4be-e03676db32c5 ONLINE 0 0 0
gptid/95e485d0-67e4-11e7-9711-e03676db32c5 ONLINE 0 0 0

errors: No known data errors
root@freenas-slot08-e9000:~ # date
Wed Sep 4 11:41:48 CEST 2019
root@freenas-slot08-e9000:~ #


Can you help me to solve this problem?

The FreeNAS is a system of big project, I don't have possibility to free action on it and I need to solve it in fast way It's lock the project
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
You can try stopping the scrub:

zpool scrub -s vol1_raidz2_freenas2
 

PCorsetti

Dabbler
Joined
Aug 28, 2019
Messages
19
Thanks for your answers.

The command that you suggested me works perfectly.

I have performed your command and the scrub task has been cancelled.

This is the status of scrub task now:
root@freenas-slot08-e9000:/mnt/vol1_raidz2_freenas2/Samba_Share # zpool status vol1_raidz2_freenas2
pool: vol1_raidz2_freenas2
state: ONLINE
scan: scrub canceled on Wed Sep 4 13:12:18 2019
config:

NAME STATE READ WRITE CKSUM
vol1_raidz2_freenas2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/926a799e-67e4-11e7-9711-e03676db32c5 ONLINE 0 0 0
gptid/93151f52-67e4-11e7-9711-e03676db32c5 ONLINE 0 0 0
gptid/93c2328a-67e4-11e7-9711-e03676db32c5 ONLINE 0 0 0
gptid/9484d9d0-67e4-11e7-9711-e03676db32c5 ONLINE 0 0 0
gptid/e76eeb0c-e1c3-11e7-b4be-e03676db32c5 ONLINE 0 0 0
gptid/95e485d0-67e4-11e7-9711-e03676db32c5 ONLINE 0 0 0

errors: No known data errors
root@freenas-slot08-e9000:/mnt/vol1_raidz2_freenas2/Samba_Share #

Unfortunately my problem has not been solved.

When I perform action on the pool as save data using rsync task the system go in stuck and it doesn't save data on disk and the rsync process go on status D.
I have supposed that my problem was connected to scrub task not completed.


Thanks for your help

 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
You may want to look at the output from dmesg to see if anything else is happening that might cause your pool to become blocked.
 

PCorsetti

Dabbler
Joined
Aug 28, 2019
Messages
19
I have verified the dmesg but I don't see problems.
Follows dmesg output:

root@freenas-slot08-e9000:/mnt/vol1_raidz2_freenas2/Samba_Share # dmesg
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.2-STABLE #0 r325575+95cc58ca2a0(HEAD): Fri May 10 15:57:35 EDT 2019
root@mp20.tn.ixsystems.com:/freenas-releng/freenas/_BE/objs/freenas-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
VT(vga): resolution 640x480
CPU: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz (2400.05-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x21<LAHF,ABM>
Structured Extended Features=0x37ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,NFPUSG>
Structured Extended Features3=0xc000000<IBPB,STIBP>
XSAVE Features=0x1<XSAVEOPT>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
TSC: P-state invariant, performance statistics
real memory = 137438953472 (131072 MB)
avail memory = 133165522944 (126996 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INSYDE HSW-LPT>
FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 hardware threads
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
SMP: AP CPU #1 Launched!
SMP: AP CPU #17 Launched!
SMP: AP CPU #28 Launched!
SMP: AP CPU #22 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #27 Launched!
SMP: AP CPU #18 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #29 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #23 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #20 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #30 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #25 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #26 Launched!
SMP: AP CPU #31 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #16 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #21 Launched!
SMP: AP CPU #24 Launched!
SMP: AP CPU #19 Launched!
Timecounter "TSC-low" frequency 1200027128 Hz quality 1000
random: entropy device external interface
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
kbd1 at kbdmux0
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
padlock0: No ACE support.
acpi0: <INSYDE HSW-LPT> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> numa-domain 0 on acpi0
cpu1: <ACPI CPU> numa-domain 0 on acpi0
cpu2: <ACPI CPU> numa-domain 0 on acpi0
cpu3: <ACPI CPU> numa-domain 0 on acpi0
cpu4: <ACPI CPU> numa-domain 0 on acpi0
cpu5: <ACPI CPU> numa-domain 0 on acpi0
cpu6: <ACPI CPU> numa-domain 0 on acpi0
cpu7: <ACPI CPU> numa-domain 0 on acpi0
cpu8: <ACPI CPU> numa-domain 0 on acpi0
cpu9: <ACPI CPU> numa-domain 0 on acpi0
cpu10: <ACPI CPU> numa-domain 0 on acpi0
cpu11: <ACPI CPU> numa-domain 0 on acpi0
cpu12: <ACPI CPU> numa-domain 0 on acpi0
cpu13: <ACPI CPU> numa-domain 0 on acpi0
cpu14: <ACPI CPU> numa-domain 0 on acpi0
cpu15: <ACPI CPU> numa-domain 0 on acpi0
cpu16: <ACPI CPU> numa-domain 1 on acpi0
cpu17: <ACPI CPU> numa-domain 1 on acpi0
cpu18: <ACPI CPU> numa-domain 1 on acpi0
cpu19: <ACPI CPU> numa-domain 1 on acpi0
cpu20: <ACPI CPU> numa-domain 1 on acpi0
cpu21: <ACPI CPU> numa-domain 1 on acpi0
cpu22: <ACPI CPU> numa-domain 1 on acpi0
cpu23: <ACPI CPU> numa-domain 1 on acpi0
cpu24: <ACPI CPU> numa-domain 1 on acpi0
cpu25: <ACPI CPU> numa-domain 1 on acpi0
cpu26: <ACPI CPU> numa-domain 1 on acpi0
cpu27: <ACPI CPU> numa-domain 1 on acpi0
cpu28: <ACPI CPU> numa-domain 1 on acpi0
cpu29: <ACPI CPU> numa-domain 1 on acpi0
cpu30: <ACPI CPU> numa-domain 1 on acpi0
cpu31: <ACPI CPU> numa-domain 1 on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
Event timer "HPET4" frequency 14318180 Hz quality 340
Event timer "HPET5" frequency 14318180 Hz quality 340
Event timer "HPET6" frequency 14318180 Hz quality 340
Event timer "HPET7" frequency 14318180 Hz quality 340
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff numa-domain 0 on acpi0
pci0: <ACPI PCI bus> numa-domain 0 on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 26 at device 1.0 numa-domain 0 on pci0
pci1: <ACPI PCI bus> numa-domain 0 on pcib1
pci1: <mass storage> at device 0.0 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> irq 32 at device 2.0 numa-domain 0 on pci0
pci2: <ACPI PCI bus> numa-domain 0 on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 32 at device 2.2 numa-domain 0 on pci0
pci3: <ACPI PCI bus> numa-domain 0 on pcib3
mps0: <Avago Technologies (LSI) SAS2308> port 0x2000-0x20ff mem 0x96840000-0x9684ffff,0x96800000-0x9683ffff irq 34 at device 0.0 numa-domain 0 on pci3
mps0: Firmware: 18.00.04.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 4985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex>
pcib4: <ACPI PCI-PCI bridge> irq 40 at device 3.0 numa-domain 0 on pci0
pci4: <ACPI PCI bus> numa-domain 0 on pcib4
oce0: <Emulex CNA NIC function:///11.0.50.0///> mem 0x9671c000-0x9671ffff,0x966c0000-0x966dffff,0x966e0000-0x966fffff irq 40 at device 0.0 numa-domain 0 on pci4
oce0: Ethernet address: e0:36:76:db:32:c5
oce0: link state changed to DOWN
oce1: <Emulex CNA NIC function:///11.0.50.0///> mem 0x96718000-0x9671bfff,0x96680000-0x9669ffff,0x966a0000-0x966bffff irq 44 at device 0.1 numa-domain 0 on pci4
oce1: Ethernet address: e0:36:76:db:32:c9
oce1: link state changed to DOWN
pci4: <serial bus, Fibre Channel> at device 0.2 (no driver attached)
pci4: <serial bus, Fibre Channel> at device 0.3 (no driver attached)
oce2: <Emulex CNA NIC function:///11.0.50.0///> mem 0x9670c000-0x9670ffff,0x965c0000-0x965dffff,0x965e0000-0x965fffff irq 40 at device 0.4 numa-domain 0 on pci4
oce2: Ethernet address: e0:36:76:db:32:c7
oce2: link state changed to DOWN
oce3: <Emulex CNA NIC function:///11.0.50.0///> mem 0x96708000-0x9670bfff,0x96580000-0x9659ffff,0x965a0000-0x965bffff irq 44 at device 0.5 numa-domain 0 on pci4
oce3: Ethernet address: e0:36:76:db:32:cb
oce3: link state changed to DOWN
oce4: <Emulex CNA NIC function:///11.0.50.0///> mem 0x96704000-0x96707fff,0x96540000-0x9655ffff,0x96560000-0x9657ffff irq 45 at device 0.6 numa-domain 0 on pci4
oce4: Ethernet address: e0:36:76:db:32:c8
oce4: link state changed to DOWN
oce5: <Emulex CNA NIC function:///11.0.50.0///> mem 0x96700000-0x96703fff,0x96500000-0x9651ffff,0x96520000-0x9653ffff irq 46 at device 0.7 numa-domain 0 on pci4
oce5: Ethernet address: e0:36:76:db:32:cc
oce5: link state changed to DOWN
pci0: <unknown> at device 17.0 (no driver attached)
ahci0: <Intel Wellsburg AHCI SATA controller> port 0x3078-0x307f,0x308c-0x308f,0x3070-0x3077,0x3088-0x308b,0x3040-0x305f mem 0x96902000-0x969027ff irq 16 at device 17.4 numa-domain 0 on pci0
ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
xhci0: <Intel Wellsburg USB 3.0 controller> mem 0x3bffff00000-0x3bffff0ffff irq 16 at device 20.0 numa-domain 0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0: waiting for BIOS to give up control
usbus0 numa-domain 0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <simple comms> at device 22.0 (no driver attached)
pci0: <simple comms> at device 22.1 (no driver attached)
ehci0: <Intel Wellsburg USB 2.0 controller> mem 0x96904000-0x969043ff irq 16 at device 26.0 numa-domain 0 on pci0
usbus1: EHCI version 1.0
usbus1 numa-domain 0 on ehci0
usbus1: 480Mbps High Speed USB v2.0
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.0 numa-domain 0 on pci0
pci5: <ACPI PCI bus> numa-domain 0 on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 28.2 numa-domain 0 on pci0
pci6: <ACPI PCI bus> numa-domain 0 on pcib6
vgapci0: <VGA-compatible display> mem 0x94000000-0x95ffffff,0x96000000-0x961fffff irq 18 at device 0.0 numa-domain 0 on pci6
vgapci0: Boot video device
pcib7: <ACPI PCI-PCI bridge> irq 19 at device 28.3 numa-domain 0 on pci0
pci7: <ACPI PCI bus> numa-domain 0 on pcib7
ehci1: <Intel Wellsburg USB 2.0 controller> mem 0x96903000-0x969033ff irq 23 at device 29.0 numa-domain 0 on pci0
usbus2: EHCI version 1.0
usbus2 numa-domain 0 on ehci1
usbus2: 480Mbps High Speed USB v2.0
isab0: <PCI-ISA bridge> at device 31.0 numa-domain 0 on pci0
isa0: <ISA bus> numa-domain 0 on isab0
ahci1: <Intel Wellsburg AHCI SATA controller> port 0x3068-0x306f,0x3084-0x3087,0x3060-0x3067,0x3080-0x3083,0x3020-0x303f mem 0x96901000-0x969017ff irq 19 at device 31.2 numa-domain 0 on pci0
ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich4: <AHCI channel> at channel 0 on ahci1
ahcich5: <AHCI channel> at channel 1 on ahci1
ahcich6: <AHCI channel> at channel 2 on ahci1
ahcich7: <AHCI channel> at channel 3 on ahci1
ahcich8: <AHCI channel> at channel 4 on ahci1
ahcich9: <AHCI channel> at channel 5 on ahci1
pcib8: <ACPI Host-PCI bridge> port 0xcf8-0xcff numa-domain 1 on acpi0
pci8: <ACPI PCI bus> numa-domain 1 on pcib8
pcib9: <ACPI PCI-PCI bridge> irq 56 at device 2.0 numa-domain 1 on pci8
pci9: <ACPI PCI bus> numa-domain 1 on pcib9
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> port 0x9020-0x903f mem 0xc8c00000-0xc8cfffff,0xc8d04000-0xc8d07fff irq 56 at device 0.0 numa-domain 1 on pci9
ix0: Using MSI-X interrupts with 9 vectors
ix0: Ethernet address: 24:4c:07:b4:16:fe
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> port 0x9000-0x901f mem 0xc8b00000-0xc8bfffff,0xc8d00000-0xc8d03fff irq 60 at device 0.1 numa-domain 1 on pci9
ix1: Using MSI-X interrupts with 9 vectors
ix1: Ethernet address: 24:4c:07:b4:16:ff
ix1: PCI Express Bus: Speed 5.0GT/s Width x8
pcib10: <ACPI PCI-PCI bridge> irq 56 at device 2.2 numa-domain 1 on pci8
pci10: <ACPI PCI bus> numa-domain 1 on pcib10
ix2: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> port 0x8020-0x803f mem 0xc8900000-0xc89fffff,0xc8a04000-0xc8a07fff irq 58 at device 0.0 numa-domain 1 on pci10
ix2: Using MSI-X interrupts with 9 vectors
ix2: Ethernet address: 24:4c:07:b4:17:00
ix2: PCI Express Bus: Speed 5.0GT/s Width x8
ix3: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> port 0x8000-0x801f mem 0xc8800000-0xc88fffff,0xc8a00000-0xc8a03fff irq 61 at device 0.1 numa-domain 1 on pci10
ix3: Using MSI-X interrupts with 9 vectors
ix3: Ethernet address: 24:4c:07:b4:17:01
ix3: PCI Express Bus: Speed 5.0GT/s Width x8
pcib11: <ACPI Host-PCI bridge> on acpi0
pci11: <ACPI PCI bus> on pcib11
pci11: <dasp, performance counters> at device 8.2 (no driver attached)
pci11: <dasp, performance counters> at device 9.2 (no driver attached)
pci11: <dasp, performance counters> at device 11.1 (no driver attached)
pci11: <dasp, performance counters> at device 11.2 (no driver attached)
pci11: <dasp, performance counters> at device 16.1 (no driver attached)
pci11: <dasp, performance counters> at device 16.6 (no driver attached)
pci11: <dasp, performance counters> at device 18.1 (no driver attached)
pcib12: <ACPI Host-PCI bridge> on acpi0
pci12: <ACPI PCI bus> on pcib12
pci12: <dasp, performance counters> at device 8.2 (no driver attached)
pci12: <dasp, performance counters> at device 9.2 (no driver attached)
pci12: <dasp, performance counters> at device 11.1 (no driver attached)
pci12: <dasp, performance counters> at device 11.2 (no driver attached)
pci12: <dasp, performance counters> at device 16.1 (no driver attached)
pci12: <dasp, performance counters> at device 16.6 (no driver attached)
pci12: <dasp, performance counters> at device 18.1 (no driver attached)
acpi_syscontainer0: <System Container> on acpi0
acpi_syscontainer1: <System Container> on acpi0
acpi_syscontainer2: <System Container> on acpi0
uart0: <16650 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
ichwd0: <Intel Wellsburg watchdog timer> on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
ichwd0: <Intel Wellsburg watchdog timer> at port 0x430-0x437,0x460-0x47f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xce000-0xcefff,0xcf000-0xcffff,0xd0000-0xd17ff,0xd1800-0xd27ff,0xd2800-0xd37ff on isa0
coretemp0: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu0
est0: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu0
coretemp1: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu1
est1: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu1
coretemp2: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu2
est2: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu2
coretemp3: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu3
est3: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu3
coretemp4: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu4
est4: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu4
coretemp5: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu5
est5: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu5
coretemp6: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu6
est6: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu6
coretemp7: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu7
est7: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu7
coretemp8: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu8
est8: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu8
coretemp9: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu9
est9: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu9
coretemp10: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu10
est10: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu10
coretemp11: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu11
est11: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu11
coretemp12: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu12
est12: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu12
coretemp13: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu13
est13: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu13
coretemp14: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu14
est14: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu14
coretemp15: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu15
est15: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu15
coretemp16: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu16
est16: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu16
coretemp17: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu17
est17: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu17
coretemp18: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu18
est18: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu18
coretemp19: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu19
est19: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu19
coretemp20: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu20
est20: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu20
coretemp21: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu21
est21: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu21
coretemp22: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu22
est22: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu22
coretemp23: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu23
est23: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu23
coretemp24: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu24
est24: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu24
coretemp25: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu25
est25: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu25
coretemp26: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu26
est26: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu26
coretemp27: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu27
est27: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu27
coretemp28: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu28
est28: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu28
coretemp29: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu29
est29: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu29
coretemp30: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu30
est30: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu30
coretemp31: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu31
est31: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu31
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
freenas_sysctl: adding account.
freenas_sysctl: adding directoryservice.
freenas_sysctl: adding middlewared.
freenas_sysctl: adding network.
freenas_sysctl: adding services.
random: unblocking device.
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ugen0.1: <0x8086 XHCI root HUB> at usbus0
ugen2.1: <Intel EHCI root HUB> at usbus2
ugen1.1: <Intel EHCI root HUB> at usbus1
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
uhub1: 2 ports with 2 removable, self powered
uhub0: 21 ports with 21 removable, self powered
uhub2: 2 ports with 2 removable, self powered
ugen1.2: <vendor 0x8087 product 0x800a> at usbus1
uhub3 numa-domain 0 on uhub1
uhub3: <vendor 0x8087 product 0x800a, class 9/0, rev 2.00/0.05, addr 2> on usbus1
ugen2.2: <vendor 0x8087 product 0x8002> at usbus2
uhub4 numa-domain 0 on uhub2
uhub4: <vendor 0x8087 product 0x8002, class 9/0, rev 2.00/0.05, addr 2> on usbus2
uhub3: 6 ports with 6 removable, self powered
uhub4: 8 ports with 8 removable, self powered
ugen2.3: <vendor 0x12d1 KeyboardMouse KVM 1.1.0> at usbus2
ukbd0 numa-domain 0 on uhub4
ukbd0: <vendor 0x12d1 KeyboardMouse KVM 1.1.0, class 0/0, rev 1.10/1.00, addr 3> on usbus2
kbd2 at ukbd0
da0 at mps0 bus 0 scbus0 target 0 lun 0
da1 at mps0 bus 0 scbus0 target 3 lun 0
da2 at mps0 bus 0 scbus0 target 4 lun 0
da3 at mps0 bus 0 scbus0 target 5 lun 0
da5 at mps0 bus 0 scbus0 target 8 lun 0
da0: <LSI Logical Volume 3000> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number 4120178070107671268
da0: 150.000MB/s transfers
da0: Command Queueing enabled
da0: 571250MB (1169920000 512 byte sectors)
da1: <SEAGATE ST4000NM0023 0004> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number Z1ZAG3E60000R630UE6Q
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 3815447MB (7814037168 512 byte sectors)
da3: da5: da2: <SEAGATE ST4000NM0023 0004> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number Z1ZAJL1C0000R630R83V
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 3815447MB (7814037168 512 byte sectors)
<SEAGATE ST4000NM0023 0004> Fixed Direct Access SPC-4 SCSI device
da6 at mps0 bus 0 scbus0 target 10 lun 0
da5: Serial Number Z1ZAG8YP0000R630UFHF
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 3815447MB (7814037168 512 byte sectors)
<SEAGATE ST4000NM0023 0004> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number Z1ZAJL060000R630R9MW
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 3815447MB (7814037168 512 byte sectors)
da6: <SEAGATE ST4000NM0023 0004> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number Z1ZAJKVT0000R630TVDN
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 3815447MB (7814037168 512 byte sectors)
da4 at mps0 bus 0 scbus0 target 6 lun 0
da4: <SEAGATE ST4000NM0023 0004> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number Z1ZAJL1G0000R630R82E
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 3815447MB (7814037168 512 byte sectors)
Trying to mount root from zfs:freenas-boot/ROOT/11.2-U4.1 []...
GEOM_MIRROR: Device mirror/swap0 launched (2/2).
GEOM_MIRROR: Device mirror/swap1 launched (2/2).
GEOM_MIRROR: Device mirror/swap2 launched (2/2).
GEOM_ELI: Device mirror/swap0.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device mirror/swap1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device mirror/swap2.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI>
ix0: link state changed to UP
ix1: link state changed to UP
ix2: link state changed to UP
lagg0: link state changed to UP
ix3: link state changed to UP
ix1: link state changed to DOWN
lagg1: link state changed to UP
ix1: link state changed to UP
ix3: link state changed to DOWN
ix3: link state changed to UP
ums0 numa-domain 0 on uhub4
ums0: <vendor 0x12d1 KeyboardMouse KVM 1.1.0, class 0/0, rev 1.10/1.00, addr 3> on usbus2
nfsd: can't register svc name
root@freenas-slot08-e9000:/mnt/vol1_raidz2_freenas2/Samba_Share #
 

PCorsetti

Dabbler
Joined
Aug 28, 2019
Messages
19
What is the output of zpool list


This is the output of zpool list:
root@freenas-slot08-e9000:/mnt/vol1_raidz2_freenas2/Samba_Share # zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
freenas-boot 556G 4.44G 552G - - - 0% 1.00x ONLINE -
vol1_raidz2_freenas2 21.8T 15.1T 6.63T - - 55% 69% 1.84x ONLINE /mnt
root@freenas-slot08-e9000:/mnt/vol1_raidz2_freenas2/Samba_Share #
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
nfsd: can't register svc name
There's one problem there, but would not be related to a pool that can't even complete a scrub.

Reboot?
 

PCorsetti

Dabbler
Joined
Aug 28, 2019
Messages
19
There's one problem there, but would not be related to a pool that can't even complete a scrub.

Reboot?

Yes, I see this message, but I agreed with you that the messages is not connected with the problem.

You suggest me to do a reboot?

For your information every day morning we are forced to do a reboot.

After cancel the scrub task I don't have rebooted the system, you suggest me to do the reboot now?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
After cancel the scrub task I don't have rebooted the system, you suggest me to do the reboot now?
If the server isn't behaving as expected and we can't see anything else wrong (we haven't looked at the logs in /var/logs yet) a reboot may be the quickest way to bring back normal operation.
 

PCorsetti

Dabbler
Joined
Aug 28, 2019
Messages
19
If the server isn't behaving as expected and we can't see anything else wrong (we haven't looked at the logs in /var/logs yet) a reboot may be the quickest way to bring back normal operation.
Now I plan to do a reboot of the system.

I have done the following grep (grep -e ERROR -e Failed -e Error -e error -e failed -e FAILED ./* > errormsg.log) to directory of /var/log to extract error from all log file (you can see the result on attached file).

For your information the system doesn't have internet access.
 

Attachments

  • errormsg_log.txt
    1.2 MB · Views: 322

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Seems your Middleware Daemon is crashing periodically, which might explain some issues, but I'm not sure the pool responsiveness is one of those.

Your server can't contact the update server at ix.

I don't see anything else recent that stands out.
 

PCorsetti

Dabbler
Joined
Aug 28, 2019
Messages
19
Thanks for your support.

I agreed with you, and I don't understand where is the problems.

As you suggested me This morning I have done the reboot of the machine.

Now I perform tests and I will you updated.
 

PCorsetti

Dabbler
Joined
Aug 28, 2019
Messages
19
Seems your Middleware Daemon is crashing periodically, which might explain some issues, but I'm not sure the pool responsiveness is one of those.

Your server can't contact the update server at ix.

I don't see anything else recent that stands out.

This is to inform you that the problems related to stuck of the pool are be removed.

Yesterday, after the reboot, I have performed 5/6 times rsync task and they are performed successfully.

Yesterday Afternoon I have enabled all rsync task scheduled and I am verifying if working properly, working in progress.

Actually, I can go on the pool directory and perform commands without problems.
I have many other problems with FreeNAS but I think that is better to open new task to have a trace.

We can tell that:

when the scrub is not completed and it is stuck all action on the pool will be performed slowly or locked, to remove this problem perform the command zpool scrub -s (pool name), suggested me by you, and perform a reboot of system.

Many thanks for your support
 

PCorsetti

Dabbler
Joined
Aug 28, 2019
Messages
19
I confirm that the problem has been solved.

After deleted the stuck scrub process using the command
Code:
zpool scrub -s vol1_raidz2_freenas2
and after the reboot of system.

Actually I have problems with rsync performances that is very slow, but this is a problem that I report in a new issue.
 

styno

Patron
Joined
Apr 11, 2016
Messages
466
Code:
root@freenas-slot08-e9000:/mnt/vol1_raidz2_freenas2/Samba_Share # zpool list
NAME                    SIZE    ALLOC   FREE    CKPOINT EXPANDSZ  FRAG  CAP  DEDUP  HEALTH  ALTROOT
freenas-boot            556G    4.44G   552G    -       -         -     0%   1.00x  ONLINE  -
vol1_raidz2_freenas2    21.8T   15.1T   6.63T   -       -         55%   69%  1.84x  ONLINE  /mnt
root@freenas-slot08-e9000:/mnt/vol1_raidz2_freenas2/Samba_Share #
DEDUP would be the first thing to try to move out of the equation when doing performance tests. Unfortunately it is not that easy to remove the deduplication from already deduplicated data in the pool.
 
Top