NAS is really slow.

[morxy49]

So this has been going on for quite a while now, maybe a year or two.
My FreeNAS started getting real slow. At first i didn't bother, but last week i decided to buy more RAM to see if it helped. It did not. Now i'm starting to get really annoyed, and i have to fix it.

My current setup is as follows:

AsRock C2550D4i
32GB ECC RAM
FreeNAS 9.3 Stable
8x3TB in RAIDz2​

The symptoms are:

• Slow in the WebGUI (everything you do takes about 10 seconds to load).
• Slow while browsing shares (can take up to a minute to open a folder or rename a file).
• Slow file transfer speeds (40-70MB/s for large files (>1GB) on a 1Gbit/s network).
  
• Slow while browsing library in Plex.
• Not slow while playing media in Plex (this is kind of weird).
• Music played in Foobar2000 can sometimes take up to 10 seconds to start playing, and then have extreme stuttering while playing. Sometimes it works fine though.
• Really slow when browsing images in Lightroom.

How can i fix this?

 

[Mirfster]

Just a few questions:

1. Did you previously enable autotune?
2. Are you running SMART tests?
3. Are you running scrubs?

 

[gpsguy]

How full is your pool?

 

[morxy49]

Just a few questions:

1. Did you previously enable autotune?
2. Are you running SMART tests?
3. Are you running scrubs?

1. No, i don't even know what that is.
2/3. Yes, they are running on a schedule. But the NAS is always slow, even when they are not running.

 

[morxy49]

How full is your pool?

Here's a picture showing all the storage

Now that i look at it, where is the remaining 1.5TiB? it seems to have dissapeared into nothing?

 

[gpsguy]

First, with the pool being 87% full, it will be slower than when it was 10% full. Do you see the warning, when you hit 80%?

Here's an exerpt on that subject from the manual

"At 90% capacity, ZFS switches from performance- to space-based optimization, which has massive performance implications. For maximum write performance and to prevent problems with drive replacement, add more capacity before a pool reaches 80%."

As to the picture, a lot of users don't understand the numbers. The first row is the raw size of your disks, including parity. So, 8x3TB is ~21.6TiB. Notice the switch in units from TB to TiB. The second row is the size of your dataset. This is the storage that you can actually use. There is some overhead. But, at the end of the day, this is what you have to work with.

Now that i look at it, where is the remaining 1.5TiB?

 

[morxy49]

First, with the pool being 87% full, it will be slower than when it was 10% full.

Here's an exerpt on that subject from the manual

"At 90% capacity, ZFS switches from performance- to space-based optimization, which has massive performance implications. For maximum write performance and to prevent problems with drive replacement, add more capacity before a pool reaches 80%."

Ok, that does make sense. However this has been going on long before my pool became this full. And i'm not even at 90% yet, so there wouldn't be any difference yet. Or am i misunderstanding?

Do you see the warning, when you hit 80%?

No, i don't recall seeing a warning? Where would that be?

As to the picture, a lot of users don't understand the numbers. The first row is the raw size of your disks, including parity. So, 8x3TB is ~21.6TiB. Notice the switch in units from TB to TiB. The second row is the size of your dataset. This is the storage that you can actually use. There is some overhead. But, at the end of the day, this is what you have to work with.

1.5TiB is for overhead!? You can't be serious? Is it really that much overhead?

 

[Mirfster]

Did you ever address/correct the issues from your other post: Critical Alert ?

 

[morxy49]

Did you ever address/correct the issues from your other post: Critical Alert ?

Yes, i replaced the bad disk, and everything was fine after that.
I do still have quite high temperatures though. Disks are somewhere between 40-50 degrees. I should fix that...

 

[gpsguy]

For autotune, see section 5.4.1 of the manual. If you did enable it, please delete the entries and disable it. If you did enable it, before you added the additional RAM, it could be a limiting factor when you add additional RAM. If I recall correctly, it's not recommended on systems with less than 64GB RAM.

While the system can be slow while scrubs, etc. are running, one of the things I (and probably Mirfster) want to know about is the health of your disks. Have your received any email messages regarding errors with the disks. We've seen forum users complain about performance, only to find out they have a degraded pool or that they have problems with one or more disks.

1. No, i don't even know what that is.
2/3. Yes, they are running on a schedule. But the NAS is always slow, even when they are not running.

 

[Mirfster]

In that case, I would tend to agree with GPSGuy at the very least you should get the space issue taken care of.
As well, posting outputs from

• zpool status
• dmesg
• smartctl -a /dev/%diskname%
  • %diskname% is shown in [Storage] - "View Disk"
    
  • Need to run it for each disk
  • Example: smartctl -a /dev/da0

Please post the output in CODE brackets. *** Easier to do this over SSH (Putty) so you can copy/paste...

 

[morxy49]

For autotune, see section 5.4.1 of the manual. If you did enable it, please delete the entries and disable it. If you did enable it, before you added the additional RAM, it could be a limiting factor when you add additional RAM. If I recall correctly, it's not recommended on systems with less than 64GB RAM.

While the system can be slow while scrubs, etc. are running, one of the things I (and probably Mirfster) want to know about is the health of your disks. Have your received any email messages regarding errors with the disks. We've seen forum users complain about performance, only to find out they have a degraded pool or that they have problems with one or more disks.

I checked it, and autotune is turned off. Since i didn't even know it existed, i highly doubt it has ever been turned on.
I have noticed that the swap partition is still at 16GB, while the RAM is now at 32GB. This shouldn't matter though, right?
I will post my disk health in the comment below.

 

[gpsguy]

For swap, the 16GB = 2GB x 8 disks. It's spread across each disk to aid with disk replacement when the drive sizes don't match up exactly.

 

[morxy49]

In that case, I would tend to agree with GPSGuy at the very least you should get the space issue taken care of.
As well, posting outputs from

• zpool status
• dmesg
• smartctl -a /dev/%diskname%
  • %diskname% is shown in [Storage] - "View Disk"
    
  • Need to run it for each disk
  • Example: smartctl -a /dev/da0

Please post the output in CODE brackets. *** Easier to do this over SSH (Putty) so you can copy/paste...

zpool status:
(yes, i know it is currently scrubing)

Code:

[root@PandorasBox] ~# zpool status
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Sat Mar  5 03:46:04 2016
config:

        NAME                                          STATE     READ WRITE CKSUM
        freenas-boot                                  ONLINE       0     0     0
          gptid/ca959912-7fe4-11e4-9f1a-d050990a6173  ONLINE       0     0     0

errors: No known data errors

  pool: pandora_vol0
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub in progress since Sat Mar  5 00:00:11 2016
        5.27T scanned out of 18.4T at 101M/s, 37h43m to go
        0 repaired, 28.65% done
config:

        NAME                                            STATE     READ WRITE CKSUM
        pandora_vol0                                    ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/b013111e-637d-11e5-b80f-d050990a6173  ONLINE       0     0     0
            gptid/12b762b6-adfd-11e3-ac90-d050990a6173  ONLINE       0     0     0
            gptid/1393a459-adfd-11e3-ac90-d050990a6173  ONLINE       0     0     0
            gptid/1473b851-adfd-11e3-ac90-d050990a6173  ONLINE       0     0     0
            gptid/155235f6-adfd-11e3-ac90-d050990a6173  ONLINE       0     0     0
            gptid/163182c3-adfd-11e3-ac90-d050990a6173  ONLINE       0     0     0
            gptid/16f4904e-adfd-11e3-ac90-d050990a6173  ONLINE       0     0     0
            gptid/17e57a41-adfd-11e3-ac90-d050990a6173  ONLINE       0     0     0

errors: No known data errors

dmesg:

Code:

[root@PandorasBox] ~# dmesg
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.3-RELEASE-p5 #0 3b4abc3: Mon Dec  8 15:09:41 PST 2014
    root@build3.ixsystems.com:/tank/home/jkh/build/M/FN/objs/os-base/amd64/fusion/jkh/M/FN/FreeBSD/src/sys/FREENAS.amd64 amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: Intel(R) Atom(TM) CPU  C2550  @ 2.41GHz (2416.73-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x406d8  Family = 0x6  Model = 0x4d  Stepping = 8
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Standard Extended Features=0x2282<TSCADJ,SMEP,ENHMOVSB>
  TSC: P-state invariant, performance statistics
real memory  = 35165044736 (33536 MB)
avail memory = 33131483136 (31596 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL  TIANO   >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  2
 cpu2 (AP): APIC ID:  4
 cpu3 (AP): APIC ID:  6
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ACPI Warning: FADT (revision 5) is longer than ACPI 5.0 version, truncating length 268 to 256 (20111123/tbfadt-325)
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ispfw: registered firmware <isp_1040>
ispfw: registered firmware <isp_1040_it>
ispfw: registered firmware <isp_1080>
ispfw: registered firmware <isp_1080_it>
ispfw: registered firmware <isp_12160>
ispfw: registered firmware <isp_12160_it>
ispfw: registered firmware <isp_2100>
ispfw: registered firmware <isp_2200>
ispfw: registered firmware <isp_2300>
ispfw: registered firmware <isp_2322>
ispfw: registered firmware <isp_2400>
ispfw: registered firmware <isp_2400_multi>
ispfw: registered firmware <isp_2500>
ispfw: registered firmware <isp_2500_multi>
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard
padlock0: No ACE support.
acpi0: <ALASKA A M I > on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> mem 0xdf740000-0xdf75ffff irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> mem 0xdf720000-0xdf73ffff irq 20 at device 3.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> mem 0xdf500000-0xdf51ffff irq 22 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> irq 23 at device 1.0 on pci3
pci4: <PCI bus> on pcib4
ahci0: <Marvell 88SE9172 AHCI SATA controller> port 0xc040-0xc047,0xc030-0xc033,0xc020-0xc027,0xc010-0xc013,0xc000-0xc00f mem 0xdf410000-0xdf4101ff irq 23 at device 0.0 on pci4
ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pcib5: <PCI-PCI bridge> irq 23 at device 5.0 on pci3
pci5: <PCI bus> on pcib5
pcib6: <PCI-PCI bridge> irq 23 at device 0.0 on pci5
pci6: <PCI bus> on pcib6
vgapci0: <VGA-compatible display> port 0xb000-0xb07f mem 0xde000000-0xdeffffff,0xdf000000-0xdf01ffff irq 23 at device 0.0 on pci6
vgapci0: Boot video device
pcib7: <PCI-PCI bridge> irq 21 at device 7.0 on pci3
pci7: <PCI bus> on pcib7
igb0: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0xa000-0xa01f mem 0xdf300000-0xdf37ffff,0xdf380000-0xdf383fff irq 21 at device 0.0 on pci7
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: bc:5f:f4:fd:a9:1c
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
pcib8: <PCI-PCI bridge> irq 23 at device 9.0 on pci3
pci8: <PCI bus> on pcib8
igb1: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0x9000-0x901f mem 0xdf200000-0xdf27ffff,0xdf280000-0xdf283fff irq 23 at device 0.0 on pci8
igb1: Using MSIX interrupts with 5 vectors
igb1: Ethernet address: bc:5f:f4:fd:a9:1d
igb1: Bound queue 0 to cpu 0
igb1: Bound queue 1 to cpu 1
igb1: Bound queue 2 to cpu 2
igb1: Bound queue 3 to cpu 3
pcib9: <ACPI PCI-PCI bridge> mem 0xdf700000-0xdf71ffff at device 4.0 on pci0
pci9: <ACPI PCI bus> on pcib9
ahci1: <Marvell 88SE9230 AHCI SATA controller> port 0xd050-0xd057,0xd040-0xd043,0xd030-0xd037,0xd020-0xd023,0xd000-0xd01f mem 0xdf610000-0xdf6107ff irq 23 at device 0.0 on pci9
ahci1: AHCI v1.20 with 8 6Gbps ports, Port Multiplier not supported
ahci1: quirks=0x900<NOBSYRES,ALTSIG>
ahcich2: <AHCI channel> at channel 0 on ahci1
ahcich3: <AHCI channel> at channel 1 on ahci1
ahcich4: <AHCI channel> at channel 2 on ahci1
ahcich5: <AHCI channel> at channel 3 on ahci1
ahcich6: <AHCI channel> at channel 4 on ahci1
ahcich7: <AHCI channel> at channel 5 on ahci1
ahcich8: <AHCI channel> at channel 6 on ahci1
ahcich9: <AHCI channel> at channel 7 on ahci1
ehci0: <Intel Avoton USB 2.0 controller> mem 0xdf763000-0xdf7633ff irq 23 at device 22.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
ahci2: <Intel Avoton AHCI SATA controller> port 0xe0d0-0xe0d7,0xe0c0-0xe0c3,0xe0b0-0xe0b7,0xe0a0-0xe0a3,0xe040-0xe05f mem 0xdf762000-0xdf7627ff irq 19 at device 23.0 on pci0
ahci2: AHCI v1.30 with 4 3Gbps ports, Port Multiplier not supported
ahcich10: <AHCI channel> at channel 0 on ahci2
ahcich11: <AHCI channel> at channel 1 on ahci2
ahcich12: <AHCI channel> at channel 2 on ahci2
ahcich13: <AHCI channel> at channel 3 on ahci2
ahci3: <Intel Avoton AHCI SATA controller> port 0xe090-0xe097,0xe080-0xe083,0xe070-0xe077,0xe060-0xe063,0xe020-0xe03f mem 0xdf761000-0xdf7617ff irq 19 at device 24.0 on pci0
ahci3: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported
ahcich14: <AHCI channel> at channel 0 on ahci3
ahcich15: <AHCI channel> at channel 1 on ahci3
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart2: <16550 or compatible> port 0x248-0x24f irq 3 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: CGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3d0-0x3db iomem 0xb8000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
fdc0: <Enhanced floppy controller> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
wbwd0: DevID 0xc3 DevRev 0x33, will not attach, please report this.
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
DUMMYNET 0xfffffe0005e30040 with IPv6 initialized (100409)
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
usbus0: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <WDC WD30EFRX-68EUZN0 80.00A80> ATA-9 SATA 3.x device
ada0: Serial Number WD-WCC4N0668998
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <WDC WD30EFRX-68EUZN0 80.00A80> ATA-9 SATA 3.x device
ada1: Serial Number WD-WCC4N0687484
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada1: quirks=0x1<4K>
ada1: Previously was known as ad6
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <WDC WD30EFRX-68EUZN0 80.00A80> ATA-9 SATA 3.x device
ada2: Serial Number WD-WMC4N1465786
ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada2: quirks=0x1<4K>
ada2: Previously was known as ad8
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <WDC WD30EFRX-68EUZN0 80.00A80> ATA-9 SATA 3.x device
ada3: Serial Number WD-WCC4N0689318
ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada3: quirks=0x1<4K>
ada3: Previously was known as ad10
ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada4: <WDC WD30EFRX-68EUZN0 80.00A80> ATA-9 SATA 3.x device
ada4: Serial Number WD-WCC4N0689564
ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada4: quirks=0x1<4K>
ada4: Previously was known as ad12
ada5 at ahcich5 bus 0 scbus5 target 0 lun 0
ada5: <WDC WD30EFRX-68EUZN0 82.00A82> ATA-9 SATA 3.x device
ada5: Serial Number WD-WCC4N2VAH9LV
ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada5: quirks=0x1<4K>
ada5: Previously was known as ad14
ada6 at ahcich14 bus 0 scbus14 target 0 lun 0
ada6: <WDC WD30EZRX-00DC0B0 80.00A80> ATA-9 SATA 3.x device
ada6: Serial Number WD-WMC1T0231244
ada6: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada6: Command Queueing enabled
ada6: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada6: quirks=0x1<4K>
ada6: Previously was known as ad32
ada7 at ahcich15 bus 0 scbus15 target 0 lun 0
ada7: <WDC WD30EZRX-00MMMB0 80.00A80> ATA-8 SATA 3.x device
ada7: Serial Number WD-WCAWZ2345552
ada7: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada7: Command Queueing enabled
ada7: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada7: quirks=0x1<4K>
ada7: Previously was known as ad34
pass6 at ahcich9 bus 0 scbus9 target 0 lun 0
pass6: <Marvell Console 1.01> Removable Processor SCSI-0 device
pass6: Serial Number HKDP221516WL
pass6: 150.000MB/s transfers (SATA 1.x, UDMA4, ATAPI 12bytes, PIO 8192bytes)
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Timecounter "TSC-low" frequency 1208362662 Hz quality 1000
Root mount waiting for: usbus0
uhub0: 8 ports with 8 removable, self powered
Root mount waiting for: usbus0
ugen0.2: <vendor 0x8087> at usbus0
uhub1: <vendor 0x8087 product 0x07db, class 9/0, rev 2.00/0.02, addr 2> on usbus0
uhub1: 4 ports with 4 removable, self powered
Root mount waiting for: usbus0
Root mount waiting for: usbus0
ugen0.3: <SanDisk> at usbus0
umass0: <SanDisk Cruzer Blade, class 0/0, rev 2.00/1.26, addr 3> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x0100
umass0:17:0:-1: Attached to scbus17
da0 at umass-sim0 bus 0 scbus17 target 0 lun 0
da0: <SanDisk Cruzer Blade 1.26> Fixed Direct Access SCSI-6 device
da0: Serial Number 2004432463189CD34009
da0: 40.000MB/s transfers
da0: 3819MB (7821312 512 byte sectors: 255H 63S/T 486C)
da0: quirks=0x2<NO_6_BYTE>
Root mount waiting for: usbus0
ugen0.4: <EATON> at usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
ugen0.5: <American Megatrends Inc.> at usbus0
uhub2: <7-port Hub> on usbus0
Root mount waiting for: usbus0
uhub2: 5 ports with 5 removable, self powered
ugen0.6: <American Megatrends Inc.> at usbus0
ukbd0: <Keyboard Interface> on usbus0
kbd2 at ukbd0
ums0: <Mouse Interface> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
Trying to mount root from zfs:freenas-boot/ROOT/default []...
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
ipmi0: <IPMI System Interface> port 0xca2,0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
ipmi0: IPMI device rev. 1, firmware rev. 0.19, version 2.0
ipmi0: Number of channels 2
ipmi0: Attached watchdog
wbwd0: DevID 0xc3 DevRev 0x33, will not attach, please report this.
GEOM_ELI: Device ada5p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device ada4p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device ada3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device ada2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device ada1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device ada0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device ada7p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device ada6p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
vboxdrv: fAsync=0 offMin=0x63b offMax=0x1016
bridge0: Ethernet address: 02:80:7a:d6:73:00
igb0: promiscuous mode enabled
bridge0: link state changed to UP
epair0a: Ethernet address: 02:c8:26:00:07:0a
epair0b: Ethernet address: 02:c8:26:00:08:0b
epair0a: link state changed to UP
epair0b: link state changed to UP
igb0: link state changed to DOWN
epair0a: promiscuous mode enabled
ng_ether_ifnet_arrival_event: can't re-name node epair0b
igb0: link state changed to UP
epair1a: Ethernet address: 02:b2:7e:00:08:0a
epair1b: Ethernet address: 02:b2:7e:00:09:0b
epair1a: link state changed to UP
epair1b: link state changed to UP
epair1a: promiscuous mode enabled
ng_ether_ifnet_arrival_event: can't re-name node epair1b
epair2a: Ethernet address: 02:56:3c:00:09:0a
epair2b: Ethernet address: 02:56:3c:00:0a:0b
epair2a: link state changed to UP
epair2b: link state changed to UP
epair2a: promiscuous mode enabled
ng_ether_ifnet_arrival_event: can't re-name node epair2b
epair3a: Ethernet address: 02:64:58:00:0a:0a
epair3b: Ethernet address: 02:64:58:00:0b:0b
epair3a: link state changed to UP
epair3b: link state changed to UP
epair3a: promiscuous mode enabled
ng_ether_ifnet_arrival_event: can't re-name node epair3b
igb0: link state changed to DOWN
igb0: link state changed to UP
igb0: link state changed to DOWN
igb0: link state changed to UP
igb0: link state changed to DOWN
igb0: link state changed to UP
igb0: link state changed to DOWN
igb0: link state changed to UP
igb0: link state changed to DOWN
igb0: link state changed to UP
igb0: link state changed to DOWN
igb0: link state changed to UP
igb0: link state changed to DOWN
igb0: link state changed to UP
igb0: link state changed to DOWN
igb0: link state changed to UP
igb0: link state changed to DOWN
igb0: link state changed to UP
arp: 192.168.1.100 moved from 02:56:3c:00:09:0a to bc:5f:f4:fd:a9:1c on epair2b
arp: 192.168.1.100 moved from 02:b2:7e:00:08:0a to bc:5f:f4:fd:a9:1c on epair1b
arp: 192.168.1.100 moved from 02:c8:26:00:07:0a to bc:5f:f4:fd:a9:1c on epair0b
arp: 192.168.1.100 moved from 02:56:3c:00:09:0a to bc:5f:f4:fd:a9:1c on epair2b
arp: 192.168.1.100 moved from 02:b2:7e:00:08:0a to bc:5f:f4:fd:a9:1c on epair1b
arp: 192.168.1.100 moved from 02:c8:26:00:07:0a to bc:5f:f4:fd:a9:1c on epair0b
[root@PandorasBox] ~#

smartctl -a /dev/xxx:
The comment became too long, so i posted it here instead: https://justpaste.it/rz31

 

[Mirfster]

Also, you can post output from arc_summary.py ; this will provide additional information. Lol, asking for so much I should just request a complete debug; but am too lazy to go extracting and parsing... :)

 

[gpsguy]

In the upper right hand corner of the webGUI. See: http://doc.freenas.org/9.3/freenas_alert.html "a volume’s capacity goes over 80%"

No, i don't recall seeing a warning? Where would that be?

BTW, you might find my PDF version of the documentation to be a useful aid. You can download it, using the link in my signature.

 

[morxy49]

In the upper right hand corner of the webGUI. See: http://doc.freenas.org/9.3/freenas_alert.html "a volume’s capacity goes over 80%"

BTW, you might find my PDF version of the documentation to be a useful aid. You can download it, using the link in my signature.

No, this is all i have.

 

[morxy49]

Also, you can post output from arc_summary.py ; this will provide additional information. Lol, asking for so much I should just request a complete debug; but am too lazy to go extracting and parsing... :)

Code:

[root@PandorasBox] ~# arc_summary.py
System Memory:

        0.57%   181.16  MiB Active,     5.08%   1.58    GiB Inact
        89.36%  27.69   GiB Wired,      0.01%   2.37    MiB Cache
        4.96%   1.54    GiB Free,       0.01%   4.10    MiB Gap

        Real Installed:                         32.00   GiB
        Real Available:                 99.84%  31.95   GiB
        Real Managed:                   96.99%  30.99   GiB

        Logical Total:                          32.00   GiB
        Logical Used:                   90.27%  28.88   GiB
        Logical Free:                   9.73%   3.12    GiB

Kernel Memory:                                  579.81  MiB
        Data:                           95.93%  556.21  MiB
        Text:                           4.07%   23.59   MiB

Kernel Memory Map:                              29.73   GiB
        Size:                           76.73%  22.81   GiB
        Free:                           23.27%  6.92    GiB
                                                                Page:  1
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Storage pool Version:                   5000
        Filesystem Version:                     5
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                1.85m
        Recycle Misses:                         275.03k
        Mutex Misses:                           167
        Evict Skips:                            167

ARC Size:                               80.54%  24.15   GiB
        Target Size: (Adaptive)         80.64%  24.18   GiB
        Min Size (Hard Limit):          12.50%  3.75    GiB
        Max Size (High Water):          8:1     29.99   GiB

ARC Size Breakdown:
        Recently Used Cache Size:       86.36%  20.88   GiB
        Frequently Used Cache Size:     13.64%  3.30    GiB

ARC Hash Breakdown:
        Elements Max:                           1.77m
        Elements Current:               99.46%  1.76m
        Collisions:                             1.07m
        Chain Max:                              6
        Chains:                                 287.64k
                                                                Page:  2
------------------------------------------------------------------------

ARC Total accesses:                                     36.19m
        Cache Hit Ratio:                91.42%  33.08m
        Cache Miss Ratio:               8.58%   3.10m
        Actual Hit Ratio:               86.36%  31.25m

        Data Demand Efficiency:         98.34%  12.76m
        Data Prefetch Efficiency:       44.19%  1.66m

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             3.09%   1.02m
          Most Recently Used:           21.78%  7.21m
          Most Frequently Used:         72.68%  24.05m
          Most Recently Used Ghost:     0.12%   40.67k
          Most Frequently Used Ghost:   2.32%   766.95k

        CACHE HITS BY DATA TYPE:
          Demand Data:                  37.92%  12.55m
          Prefetch Data:                2.21%   732.71k
          Demand Metadata:              54.63%  18.07m
          Prefetch Metadata:            5.23%   1.73m

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  6.80%   211.22k
          Prefetch Data:                29.80%  925.25k
          Demand Metadata:              43.79%  1.36m
          Prefetch Metadata:            19.61%  608.87k
                                                                Page:  3
------------------------------------------------------------------------

                                                                Page:  4
------------------------------------------------------------------------

File-Level Prefetch: (HEALTHY)
DMU Efficiency:                                 54.24m
        Hit Ratio:                      55.95%  30.35m
        Miss Ratio:                     44.05%  23.90m

        Colinear:                               23.90m
          Hit Ratio:                    0.03%   7.60k
          Miss Ratio:                   99.97%  23.89m

        Stride:                                 28.27m
          Hit Ratio:                    99.86%  28.23m
          Miss Ratio:                   0.14%   39.18k

DMU Misc:
        Reclaim:                                23.89m
          Successes:                    1.91%   456.05k
          Failures:                     98.09%  23.43m

        Streams:                                2.12m
          +Resets:                      0.04%   797
          -Resets:                      99.96%  2.12m
          Bogus:                                0
                                                                Page:  5
------------------------------------------------------------------------

                                                                Page:  6
------------------------------------------------------------------------

ZFS Tunable (sysctl):
        kern.maxusers                           2380
        vm.kmem_size                            33272016896
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        329853485875
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_lsize            15054321152
        vfs.zfs.mfu_ghost_metadata_lsize        5843885056
        vfs.zfs.mfu_ghost_size                  20898206208
        vfs.zfs.mfu_data_lsize                  664062976
        vfs.zfs.mfu_metadata_lsize              2281228800
        vfs.zfs.mfu_size                        3298514432
        vfs.zfs.mru_ghost_data_lsize            5044706816
        vfs.zfs.mru_ghost_metadata_lsize        20819456
        vfs.zfs.mru_ghost_size                  5065526272
        vfs.zfs.mru_data_lsize                  17218535424
        vfs.zfs.mru_metadata_lsize              2870104064
        vfs.zfs.mru_size                        20409341952
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       334848
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  8049568768
        vfs.zfs.arc_meta_used                   8049563112
        vfs.zfs.arc_shrink_shift                5
        vfs.zfs.arc_average_blocksize           8192
        vfs.zfs.arc_min                         4024784384
        vfs.zfs.arc_max                         32198275072
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.nopwrite_enabled                1
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                0
        vfs.zfs.delay_scale                     500000
        vfs.zfs.delay_min_dirty_percent         60
        vfs.zfs.dirty_data_sync                 67108864
        vfs.zfs.dirty_data_max_percent          10
        vfs.zfs.dirty_data_max_max              4294967296
        vfs.zfs.dirty_data_max                  3430471680
        vfs.zfs.free_max_blocks                 131072
        vfs.zfs.no_scrub_prefetch               0
        vfs.zfs.no_scrub_io                     0
        vfs.zfs.resilver_min_time_ms            3000
        vfs.zfs.free_min_time_ms                1000
        vfs.zfs.scan_min_time_ms                1000
        vfs.zfs.scan_idle                       50
        vfs.zfs.scrub_delay                     4
        vfs.zfs.resilver_delay                  2
        vfs.zfs.top_maxinflight                 32
        vfs.zfs.mg_fragmentation_threshold      85
        vfs.zfs.mg_noalloc_threshold            0
        vfs.zfs.condense_pct                    200
        vfs.zfs.metaslab.bias_enabled           1
        vfs.zfs.metaslab.lba_weighting_enabled  1
        vfs.zfs.metaslab.fragmentation_factor_enabled1
        vfs.zfs.metaslab.preload_enabled        1
        vfs.zfs.metaslab.preload_limit          3
        vfs.zfs.metaslab.unload_delay           8
        vfs.zfs.metaslab.load_pct               50
        vfs.zfs.metaslab.min_alloc_size         33554432
        vfs.zfs.metaslab.df_free_pct            4
        vfs.zfs.metaslab.df_alloc_threshold     131072
        vfs.zfs.metaslab.debug_unload           0
        vfs.zfs.metaslab.debug_load             0
        vfs.zfs.metaslab.fragmentation_threshold70
        vfs.zfs.metaslab.gang_bang              16777217
        vfs.zfs.spa_load_verify_data            1
        vfs.zfs.spa_load_verify_metadata        1
        vfs.zfs.spa_load_verify_maxinflight     10000
        vfs.zfs.ccw_retry_interval              300
        vfs.zfs.check_hostid                    1
        vfs.zfs.spa_asize_inflation             24
        vfs.zfs.deadman_enabled                 1
        vfs.zfs.deadman_checktime_ms            5000
        vfs.zfs.deadman_synctime_ms             1000000
        vfs.zfs.recover                         0
        vfs.zfs.space_map_blksz                 32768
        vfs.zfs.trim.max_interval               1
        vfs.zfs.trim.timeout                    30
        vfs.zfs.trim.txg_delay                  32
        vfs.zfs.trim.enabled                    1
        vfs.zfs.txg.timeout                     5
        vfs.zfs.min_auto_ashift                 9
        vfs.zfs.max_auto_ashift                 13
        vfs.zfs.vdev.trim_max_pending           64
        vfs.zfs.vdev.trim_max_bytes             2147483648
        vfs.zfs.vdev.metaslabs_per_vdev         200
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.larger_ashift_minimal      0
        vfs.zfs.vdev.bio_delete_disable         0
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.vdev.trim_on_init               1
        vfs.zfs.vdev.mirror.non_rotating_seek_inc1
        vfs.zfs.vdev.mirror.non_rotating_inc    0
        vfs.zfs.vdev.mirror.rotating_seek_offset1048576
        vfs.zfs.vdev.mirror.rotating_seek_inc   5
        vfs.zfs.vdev.mirror.rotating_inc        0
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.trim_max_active            64
        vfs.zfs.vdev.trim_min_active            1
        vfs.zfs.vdev.scrub_max_active           2
        vfs.zfs.vdev.scrub_min_active           1
        vfs.zfs.vdev.async_write_max_active     10
        vfs.zfs.vdev.async_write_min_active     1
        vfs.zfs.vdev.async_read_max_active      3
        vfs.zfs.vdev.async_read_min_active      1
        vfs.zfs.vdev.sync_write_max_active      10
        vfs.zfs.vdev.sync_write_min_active      10
        vfs.zfs.vdev.sync_read_max_active       10
        vfs.zfs.vdev.sync_read_min_active       10
        vfs.zfs.vdev.max_active                 1000
        vfs.zfs.vdev.async_write_active_max_dirty_percent60
        vfs.zfs.vdev.async_write_active_min_dirty_percent30
        vfs.zfs.snapshot_list_prefetch          0
        vfs.zfs.version.ioctl                   4
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     5000
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.sync_pass_rewrite               2
        vfs.zfs.sync_pass_dont_compress         5
        vfs.zfs.sync_pass_deferred_free         2
        vfs.zfs.zio.use_uma                     1
        vfs.zfs.vol.unmap_enabled               1
        vfs.zfs.vol.mode                        2
                                                                Page:  7
------------------------------------------------------------------------

 

[Mirfster]

This is all I see of concern from the SMART results (please correct me if I am wrong).
dev/ada6

Code:

197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 1
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 2

 

[gpsguy]

I saw those too.

The LCC counts on the 2 Greens are quite high.

This is all I see of concern from the SMART results