SOLVED Critical alert, how best to proceed?

[nasnice]

Hi after 3 months of flawless performance I now have a serious? problem.

I receive these mails:
"
The volume Z1 (ZFS) state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.
"
and:
"
host name: Nas
DNS domain: workgroup

The following warning/error was logged by the smartd daemon:

Device: /dev/ada1, 56 Currently unreadable (pending) sectors (changed +8)

Device info:
ST3000DM001-9YN166, S/N:Z1F16SA3, WWN:5-000c50-04e53d348, FW:CC4H, 3.00 TB

For details see host's SYSLOG.

You can also use the smartctl utility for further investigation.
No additional messages about this problem will be sent.
"
and
"
This message was generated by the smartd daemon running on:

host name: Nas
DNS domain: workgroup

The following warning/error was logged by the smartd daemon:

Device: /dev/ada1, 56 Offline uncorrectable sectors (changed +8)

Device info:
ST3000DM001-9YN166, S/N:Z1F16SA3, WWN:5-000c50-04e53d348, FW:CC4H, 3.00 TB

For details see host's SYSLOG.

You can also use the smartctl utility for further investigation.
No additional messages about this problem will be sent.
"
Is this meant by Syslog?
"

Code:

May  3 14:39:57 Nas (ada1:ahcich1:0:0:0): Retrying command
May  3 14:39:57 Nas ahcich1: Error while READ LOG EXT
May  3 14:39:57 Nas (ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 86 a3 50 40 5d 01 00 00 00 00
May  3 14:39:57 Nas (ada1:ahcich1:0:0:0): CAM status: ATA Status Error
May  3 14:39:57 Nas (ada1:ahcich1:0:0:0): ATA status: 00 ()
May  3 14:39:57 Nas (ada1:ahcich1:0:0:0): RES: 00 00 00 00 00 00 00 00 00 00 00
May  3 14:39:57 Nas (ada1:ahcich1:0:0:0): Retrying command
May  3 14:39:57 Nas ahcich1: Error while READ LOG EXT
May  3 14:39:57 Nas (ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 86 a3 50 40 5d 01 00 00 00 00
//
May  3 14:39:57 Nas (ada1:ahcich1:0:0:0): Error 5, Retries exhausted
May  3 14:40:56 Nas smbd[2389]: dnssd_clientstub DNSServiceProcessResult called with DNSServiceRef with no ProcessReply function
May  3 14:40:58 Nas winbindd[2392]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=1)
May  3 14:40:58 Nas winbindd[2397]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=0)
May  3 14:40:58 Nas winbindd[2398]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=0)
May  3 14:40:58 Nas winbindd[2396]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=0)
May  3 14:40:58 Nas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
May  3 14:40:59 Nas nmbd[2386]:   STATUS=daemon 'nmbd' finished starting up and ready to serve connectionsGot SIGTERM: going down...
May  3 14:40:59 Nas upsmon[2235]: upsmon parent: read
May  3 14:40:59 Nas upsd[2229]: mainloop: Interrupted system call
May  3 14:40:59 Nas ntpd[2117]: ntpd exiting on signal 15
May  3 14:40:59 Nas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
May  3 14:41:00 Nas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
May  3 14:41:01 Nas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused
May  3 14:41:04 Nas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
May  3 14:41:05 Nas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
May  3 14:41:05 Nas syslog-ng[1749]: syslog-ng shutting down; version='3.5.6'
May  3 14:44:33 Nas syslog-ng[1748]: syslog-ng starting up; version='3.5.6'
May  3 14:44:33 Nas Copyright (c) 1992-2014 The FreeBSD Project.
May  3 14:44:33 Nas Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
May  3 14:44:33 Nas     The Regents of the University of California. All rights reserved.
May  3 14:44:33 Nas FreeBSD is a registered trademark of The FreeBSD Foundation.
May  3 14:44:33 Nas FreeBSD 9.3-RELEASE-p8 #1 r275790+18ab2bc: Wed Jan 21 12:32:31 PST 2015
May  3 14:44:33 Nas root@build3.ixsystems.com:/tank/home/jkh/build/93/FN/objs/os-base/amd64/fusion/jkh/93/FN/FreeBSD/src/sys/FREENAS.amd64 amd64
May  3 14:44:33 Nas gcc version 4.2.1 20070831 patched [FreeBSD]
May  3 14:44:33 Nas CPU: Intel(R) Celeron(R) CPU G1820 @ 2.70GHz (2699.27-MHz K8-class CPU)
May  3 14:44:33 Nas Origin = "GenuineIntel"  Id = 0x306c3  Family = 0x6  Model = 0x3c  Stepping = 3
May  3 14:44:33 Nas Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
May  3 14:44:33 Nas Features2=0x4ddaebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,<b11>,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,XSAVE,OSXSAVE,RDRAND>
May  3 14:44:33 Nas AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
May  3 14:44:33 Nas AMD Features2=0x21<LAHF,ABM>
May  3 14:44:33 Nas Standard Extended Features=0x2603<GSFSBASE,TSCADJ,ENHMOVSB,INVPCID>
May  3 14:44:33 Nas TSC: P-state invariant, performance statistics
May  3 14:44:33 Nas real memory  = 17689477120 (16870 MB)
May  3 14:44:33 Nas avail memory = 16165195776 (15416 MB)
May  3 14:44:33 Nas Event timer "LAPIC" quality 600
May  3 14:44:33 Nas ACPI APIC Table: <ALASKA A M I>
May  3 14:44:33 Nas FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
May  3 14:44:33 Nas FreeBSD/SMP: 1 package(s) x 2 core(s)
May  3 14:44:33 Nas cpu0 (BSP): APIC ID:  0
May  3 14:44:33 Nas cpu1 (AP): APIC ID:  2
May  3 14:44:33 Nas WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
May  3 14:44:33 Nas ACPI Warning: FADT (revision 5) is longer than ACPI 5.0 version, truncating length 268 to 256 (20111123/tbfadt-325)
May  3 14:44:33 Nas ioapic0 <Version 2.0> irqs 0-23 on motherboard
May  3 14:44:33 Nas ispfw: registered firmware <isp_1040>
May  3 14:44:33 Nas ispfw: registered firmware <isp_1040_it>
May  3 14:44:33 Nas ispfw: registered firmware <isp_1080>
May  3 14:44:33 Nas ispfw: registered firmware <isp_1080_it>
May  3 14:44:33 Nas ispfw: registered firmware <isp_12160>
May  3 14:44:33 Nas ispfw: registered firmware <isp_12160_it>
May  3 14:44:33 Nas ispfw: registered firmware <isp_2100>
May  3 14:44:33 Nas ispfw: registered firmware <isp_2200>
May  3 14:44:33 Nas ispfw: registered firmware <isp_2300>
May  3 14:44:33 Nas ispfw: registered firmware <isp_2322>
May  3 14:44:33 Nas ispfw: registered firmware <isp_2400>
May  3 14:44:33 Nas ispfw: registered firmware <isp_2400_multi>
May  3 14:44:33 Nas ispfw: registered firmware <isp_2500>
May  3 14:44:33 Nas ispfw: registered firmware <isp_2500_multi>
May  3 14:44:33 Nas kbd1 at kbdmux0
May  3 14:44:33 Nas cryptosoft0: <software crypto> on motherboard
May  3 14:44:33 Nas aesni0: No AESNI support.
May  3 14:44:33 Nas padlock0: No ACE support.
May  3 14:44:33 Nas acpi0: <ALASKA A M I> on motherboard
May  3 14:44:33 Nas acpi0: Power Button (fixed)
May  3 14:44:33 Nas acpi0: reservation of 67, 1 (4) failed
May  3 14:44:33 Nas cpu0: <ACPI CPU> on acpi0
May  3 14:44:33 Nas cpu1: <ACPI CPU> on acpi0
May  3 14:44:33 Nas hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
May  3 14:44:33 Nas Timecounter "HPET" frequency 14318180 Hz quality 950
May  3 14:44:33 Nas Event timer "HPET" frequency 14318180 Hz quality 550
May  3 14:44:33 Nas Event timer "HPET1" frequency 14318180 Hz quality 440
May  3 14:44:33 Nas Event timer "HPET2" frequency 14318180 Hz quality 440
May  3 14:44:33 Nas Event timer "HPET3" frequency 14318180 Hz quality 440
May  3 14:44:33 Nas Event timer "HPET4" frequency 14318180 Hz quality 440
May  3 14:44:33 Nas Event timer "HPET5" frequency 14318180 Hz quality 440
May  3 14:44:33 Nas Event timer "HPET6" frequency 14318180 Hz quality 440
May  3 14:44:33 Nas atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
May  3 14:44:33 Nas atrtc0: Warning: Couldn't map I/O.
May  3 14:44:33 Nas Event timer "RTC" frequency 32768 Hz quality 0
May  3 14:44:33 Nas attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
May  3 14:44:33 Nas Timecounter "i8254" frequency 1193182 Hz quality 0
May  3 14:44:33 Nas Event timer "i8254" frequency 1193182 Hz quality 100
May  3 14:44:33 Nas Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
May  3 14:44:33 Nas acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
May  3 14:44:33 Nas pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
May  3 14:44:33 Nas pci0: <ACPI PCI bus> on pcib0
May  3 14:44:33 Nas vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xf0000000-0xf03fffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0
May  3 14:44:33 Nas vgapci0: Boot video device
May  3 14:44:33 Nas pci0: <multimedia, HDA> at device 3.0 (no driver attached)
May  3 14:44:33 Nas pci0: <serial bus, USB> at device 20.0 (no driver attached)
May  3 14:44:33 Nas pci0: <simple comms> at device 22.0 (no driver attached)
May  3 14:44:33 Nas em0: <Intel(R) PRO/1000 Network Connection 7.4.2> port 0xf080-0xf09f mem 0xf0400000-0xf041ffff,0xf043d000-0xf043dfff irq 20 at device 25.0 on pci0
May  3 14:44:33 Nas em0: Using an MSI interrupt
May  3 14:44:33 Nas em0: Ethernet address: d0:50:99:0d:b2:f5
May  3 14:44:33 Nas ehci0: <EHCI (generic) USB 2.0 controller> mem 0xf043c000-0xf043c3ff irq 16 at device 26.0 on pci0
May  3 14:44:33 Nas usbus0: EHCI version 1.0
May  3 14:44:33 Nas usbus0 on ehci0
May  3 14:44:33 Nas pci0: <multimedia, HDA> at device 27.0 (no driver attached)
May  3 14:44:33 Nas pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
May  3 14:44:33 Nas pci1: <ACPI PCI bus> on pcib1
May  3 14:44:33 Nas pcib2: <ACPI PCI-PCI bridge> at device 28.3 on pci0
May  3 14:44:33 Nas pci2: <ACPI PCI bus> on pcib2
May  3 14:44:33 Nas pcib3: <ACPI PCI-PCI bridge> irq 19 at device 0.0 on pci2
May  3 14:44:33 Nas pci3: <ACPI PCI bus> on pcib3
May  3 14:44:33 Nas ehci1: <EHCI (generic) USB 2.0 controller> mem 0xf043b000-0xf043b3ff irq 23 at device 29.0 on pci0
May  3 14:44:33 Nas usbus1: EHCI version 1.0
May  3 14:44:33 Nas usbus1 on ehci1
May  3 14:44:33 Nas isab0: <PCI-ISA bridge> at device 31.0 on pci0
May  3 14:44:33 Nas isa0: <ISA bus> on isab0
May  3 14:44:33 Nas ahci0: <Intel Lynx Point AHCI SATA controller> port 0xf0d0-0xf0d7,0xf0c0-0xf0c3,0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf060-0xf07f mem 0xf043a000-0xf043a7ff irq 19 at device 31.2 on pci0
May  3 14:44:33 Nas ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
May  3 14:44:33 Nas ahcich0: <AHCI channel> at channel 0 on ahci0
May  3 14:44:33 Nas ahcich1: <AHCI channel> at channel 1 on ahci0
May  3 14:44:33 Nas ahcich2: <AHCI channel> at channel 2 on ahci0
May  3 14:44:33 Nas ahcich3: <AHCI channel> at channel 3 on ahci0
May  3 14:44:33 Nas ahcich4: <AHCI channel> at channel 4 on ahci0
May  3 14:44:33 Nas ahcich5: <AHCI channel> at channel 5 on ahci0
May  3 14:44:33 Nas acpi_button0: <Power Button> on acpi0
May  3 14:44:33 Nas sc0: <System console> at flags 0x100 on isa0
May  3 14:44:33 Nas sc0: VGA <16 virtual consoles, flags=0x300>
May  3 14:44:33 Nas vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
May  3 14:44:33 Nas atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
May  3 14:44:33 Nas atkbd0: <AT Keyboard> irq 1 on atkbdc0
May  3 14:44:33 Nas kbd0 at atkbd0
May  3 14:44:33 Nas atkbd0: [GIANT-LOCKED]
May  3 14:44:33 Nas wbwd0: DevID 0xc3 DevRev 0x33, will not attach, please report this.
May  3 14:44:33 Nas coretemp0: <CPU On-Die Thermal Sensors> on cpu0
May  3 14:44:33 Nas est0: <Enhanced SpeedStep Frequency Control> on cpu0
May  3 14:44:33 Nas p4tcc0: <CPU Frequency Thermal Control> on cpu0
May  3 14:44:33 Nas coretemp1: <CPU On-Die Thermal Sensors> on cpu1
May  3 14:44:33 Nas est1: <Enhanced SpeedStep Frequency Control> on cpu1
May  3 14:44:33 Nas p4tcc1: <CPU Frequency Thermal Control> on cpu1
May  3 14:44:33 Nas ZFS filesystem version: 5
May  3 14:44:33 Nas ZFS storage pool version: features support (5000)
May  3 14:44:33 Nas Timecounters tick every 1.000 msec
May  3 14:44:33 Nas ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
May  3 14:44:33 Nas usbus0: 480Mbps High Speed USB v2.0
May  3 14:44:33 Nas usbus1: 480Mbps High Speed USB v2.0
May  3 14:44:33 Nas ugen0.1: <Intel> at usbus0
May  3 14:44:33 Nas uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
May  3 14:44:33 Nas ugen1.1: <Intel> at usbus1
May  3 14:44:33 Nas uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
May  3 14:44:33 Nas uhub0: 2 ports with 2 removable, self powered
May  3 14:44:33 Nas uhub1: 2 ports with 2 removable, self powered
May  3 14:44:33 Nas ugen0.2: <vendor 0x8087> at usbus0
May  3 14:44:33 Nas uhub2: <vendor 0x8087 product 0x8008, class 9/0, rev 2.00/0.05, addr 2> on usbus0
May  3 14:44:33 Nas ugen1.2: <vendor 0x8087> at usbus1
May  3 14:44:33 Nas uhub3: <vendor 0x8087 product 0x8000, class 9/0, rev 2.00/0.05, addr 2> on usbus1
May  3 14:44:33 Nas uhub2: 6 ports with 6 removable, self powered
May  3 14:44:33 Nas uhub3: 6 ports with 6 removable, self powered
May  3 14:44:33 Nas ugen0.3: <Dell> at usbus0
May  3 14:44:33 Nas ukbd0: <Dell Dell USB Keyboard, class 0/0, rev 1.10/3.01, addr 3> on usbus0
May  3 14:44:33 Nas kbd2 at ukbd0
May  3 14:44:33 Nas ugen1.3: <USBest Technology> at usbus1
May  3 14:44:33 Nas umass0: <USBest Technology USB Mass Storage Device, class 0/0, rev 2.00/1.00, addr 3> on usbus1
May  3 14:44:33 Nas umass0:  SCSI over Bulk-Only; quirks = 0x0000
May  3 14:44:33 Nas umass0:7:0:-1: Attached to scbus7
May  3 14:44:33 Nas ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
May  3 14:44:33 Nas ada0: <WDC WD30EFRX-68EUZN0 80.00A80> ATA-9 SATA 3.x device
May  3 14:44:33 Nas ada0: Serial Number WD-WCC4N1079793
May  3 14:44:33 Nas da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
May  3 14:44:33 Nas da0: <Ut190 USB2FlashStorage 0.00> Removable Direct Access SCSI-2 device
May  3 14:44:33 Nas da0: Serial Number 00000000000232
May  3 14:44:33 Nas da0: 40.000MB/s transfers
May  3 14:44:33 Nas da0: 7712MB (15794176 512 byte sectors: 255H 63S/T 983C)
May  3 14:44:33 Nas da0: quirks=0x2<NO_6_BYTE>
May  3 14:44:33 Nas ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
May  3 14:44:33 Nas ada0: Command Queueing enabled
May  3 14:44:33 Nas ada0: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
May  3 14:44:33 Nas ada0: quirks=0x1<4K>
May  3 14:44:33 Nas ada0: Previously was known as ad4
May  3 14:44:33 Nas ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
May  3 14:44:33 Nas ada1: <ST3000DM001-9YN166 CC4H> ATA-8 SATA 3.x device
May  3 14:44:33 Nas ada1: Serial Number Z1F16SA3
May  3 14:44:33 Nas ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
May  3 14:44:33 Nas ada1: Command Queueing enabled
May  3 14:44:33 Nas ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
May  3 14:44:33 Nas ada1: quirks=0x1<4K>
May  3 14:44:33 Nas ada1: Previously was known as ad6
May  3 14:44:33 Nas ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
May  3 14:44:33 Nas ada2: <ST3000DM001-1CH166 CC29> ATA-9 SATA 3.x device
May  3 14:44:33 Nas ada2: Serial Number Z1F4R09Z
May  3 14:44:33 Nas ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
May  3 14:44:33 Nas ada2: Command Queueing enabled
May  3 14:44:33 Nas ada2: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
May  3 14:44:33 Nas ada2: quirks=0x1<4K>
May  3 14:44:33 Nas ada2: Previously was known as ad8
May  3 14:44:33 Nas ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
May  3 14:44:33 Nas ada3: <ST3000DM001-9YN166 CC4H> ATA-8 SATA 3.x device
May  3 14:44:33 Nas ada3: Serial Number Z1F16S80
May  3 14:44:33 Nas ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
May  3 14:44:33 Nas ada3: Command Queueing enabled
May  3 14:44:33 Nas ada3: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
May  3 14:44:33 Nas ada3: quirks=0x1<4K>
May  3 14:44:33 Nas ada3: Previously was known as ad10
May  3 14:44:33 Nas ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
May  3 14:44:33 Nas ada4: <WDC WD30EFRX-68EUZN0 80.00A80> ATA-9 SATA 3.x device
May  3 14:44:33 Nas ada4: Serial Number WD-WMC4N0D5W4K3
May  3 14:44:33 Nas ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
May  3 14:44:33 Nas da1 at umass-sim0 bus 0 scbus7 target 0 lun 1
May  3 14:44:33 Nas da1: <Ut190 SD0StorageDevice 0.00> Removable Direct Access SCSI-2 device
May  3 14:44:33 Nas da1: Serial Number 00000000000232
May  3 14:44:33 Nas da1: 40.000MB/s transfers
May  3 14:44:33 Nas da1: Attempt to query device size failed: NOT READY, Medium not present
May  3 14:44:33 Nas da1: quirks=0x2<NO_6_BYTE>
May  3 14:44:33 Nas ada4: Command Queueing enabled
May  3 14:44:33 Nas ada4: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
May  3 14:44:33 Nas ada4: quirks=0x1<4K>
May  3 14:44:33 Nas ada4: Previously was known as ad12
May  3 14:44:33 Nas ada5 at ahcich5 bus 0 scbus5 target 0 lun 0
May  3 14:44:33 Nas ada5: <WDC WD30EFRX-68EUZN0 80.00A80> ATA-9 SATA 3.x device
May  3 14:44:33 Nas ada5: Serial Number WD-WMC4N0D4CV3D
May  3 14:44:33 Nas ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
May  3 14:44:33 Nas ada5: Command Queueing enabled
May  3 14:44:33 Nas ada5: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
May  3 14:44:33 Nas ada5: quirks=0x1<4K>
May  3 14:44:33 Nas ada5: Previously was known as ad14
May  3 14:44:33 Nas SMP: AP CPU #1 Launched!
May  3 14:44:33 Nas Timecounter "TSC-low" frequency 1349634358 Hz quality 1000
May  3 14:44:33 Nas Trying to mount root from zfs:freenas-boot/ROOT/Initial-Install []...
May  3 14:44:33 Nas GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
May  3 14:44:33 Nas wbwd0: DevID 0xc3 DevRev 0x33, will not attach, please report this.
May  3 14:44:34 Nas root: /etc/rc: WARNING: failed to start watchdogd
May  3 14:44:36 Nas GEOM_ELI: Device ada0p1.eli created.
May  3 14:44:36 Nas GEOM_ELI: Encryption: AES-XTS 256
May  3 14:44:36 Nas GEOM_ELI:     Crypto: software
May  3 14:44:36 Nas GEOM_ELI: Device ada1p1.eli created.
May  3 14:44:36 Nas GEOM_ELI: Encryption: AES-XTS 256
May  3 14:44:36 Nas GEOM_ELI:     Crypto: software
May  3 14:44:36 Nas GEOM_ELI: Device ada2p1.eli created.
May  3 14:44:36 Nas GEOM_ELI: Encryption: AES-XTS 256
May  3 14:44:36 Nas GEOM_ELI:     Crypto: software
May  3 14:44:36 Nas GEOM_ELI: Device ada3p1.eli created.
May  3 14:44:36 Nas GEOM_ELI: Encryption: AES-XTS 256
May  3 14:44:36 Nas GEOM_ELI:     Crypto: software
May  3 14:44:36 Nas GEOM_ELI: Device ada4p1.eli created.
May  3 14:44:36 Nas GEOM_ELI: Encryption: AES-XTS 256
May  3 14:44:36 Nas GEOM_ELI:     Crypto: software
May  3 14:44:36 Nas GEOM_ELI: Device ada5p1.eli created.
May  3 14:44:36 Nas GEOM_ELI: Encryption: AES-XTS 256
May  3 14:44:36 Nas GEOM_ELI:     Crypto: software
May  3 14:44:38 Nas root: /etc/rc: WARNING: failed precmd routine for vmware_guestd
May  3 14:44:40 Nas vboxdrv: fAsync=0 offMin=0x2aa offMax=0xd2a
May  3 14:44:42 Nas ntpd[2115]: ntpd 4.2.4p5-a (1)
May  3 14:44:51 Nas generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: /sbin/sysctl -n 'kern.maxfilesperproc'
May  3 14:44:51 Nas generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: zfs list -H -o mountpoint,name
May  3 14:44:51 Nas generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: zfs list -H -o mountpoint
May  3 14:44:51 Nas ntpd[2116]: time reset -0.558002 s
May  3 14:44:51 Nas generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: zfs list -H -o mountpoint
May  3 14:44:51 Nas generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: zfs list -H -o mountpoint
May  3 14:44:51 Nas generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: zfs list -H -o mountpoint
May  3 14:44:51 Nas generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: zfs list -H -o mountpoint
May  3 14:44:51 Nas generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: zfs list -H -o mountpoint
May  3 14:44:51 Nas generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: /usr/local/bin/pdbedit -d 0 -i smbpasswd:/tmp/tmpbCSKQd -s /usr/local/etc/smb4.conf -e tdbsam:/var/etc/private/passdb.tdb
May  3 14:44:52 Nas generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: /usr/local/bin/pdbedit -L
May  3 14:44:53 Nas generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: /usr/local/bin/net sam rights grant Mover SeTakeOwnershipPrivilege SeBackupPrivilege SeRestorePrivilege
May  3 14:44:53 Nas nmbd[2385]: [2015/05/03 14:44:53.396580,  0] ../lib/util/become_daemon.c:136(daemon_ready)
May  3 14:44:53 Nas winbindd[2391]: [2015/05/03 14:44:53.546883,  0] ../source3/winbindd/winbindd_cache.c:3196(initialize_winbindd_cache)
May  3 14:44:53 Nas winbindd[2391]:   initialize_winbindd_cache: clearing cache and re-creating with version number 2
May  3 14:44:53 Nas winbindd[2391]: [2015/05/03 14:44:53.555671,  0] ../lib/util/become_daemon.c:136(daemon_ready)
May  3 14:44:53 Nas smbd[2388]: [2015/05/03 14:44:53.888081,  0] ../lib/util/become_daemon.c:136(daemon_ready)
May  3 14:44:53 Nas smbd[2388]: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
May  3 14:44:54 Nas smartd[2394]: Device: /dev/ada1, 56 Currently unreadable (pending) sectors
May  3 14:44:54 Nas smartd[2394]: Device: /dev/ada1, 56 Offline uncorrectable sectors
May  3 14:44:54 Nas smbd[2388]: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
May  3 14:44:55 Nas smbd[2388]: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
May  3 15:11:09 Nas notifier: Stopping smartd.
May  3 15:11:09 Nas notifier: Waiting for PIDS: 2399.
May  3 15:11:09 Nas notifier: smartd not running? (check /var/run/smartd.pid).
May  3 15:11:09 Nas notifier: Starting smartd.
May  3 15:11:10 Nas smartd[5415]: Device: /dev/ada1, 56 Currently unreadable (pending) sectors
May  3 15:11:10 Nas smartd[5415]: Device: /dev/ada1, 56 Offline uncorrectable sectors
May  3 15:11:40 Nas notifier: Stopping smartd.
May  3 15:11:40 Nas notifier: Waiting for PIDS: 5418.
May  3 15:11:40 Nas notifier: smartd not running? (check /var/run/smartd.pid).
May  3 15:11:40 Nas notifier: Starting smartd.

I Think one of my Seagates is about to die I have another WD red standing by. What is the best way to replace the failing Seagate disk?

 

[alexg]

Post "zpool status" and "smartctl -a /dev/ada1" results

 

[Bidule0hm]

Seems like one drive (ada1) has decided to start to fail... usually not a big deal but with a RAID-Z1 you should do a backup right now if you don't already have one and change the drive ASAP (after we confirmed it's the drive fault with the outputs of the commands @alexg asked) because one failing drive is all a RAID-Z1 allows.

 

[nasnice]

Hi alexg and BiduleOhm,

As requested:
"

Code:

zpool status                                                                                                        
  pool: Z1                                                                                                                        
state: ONLINE                                                                                                                    
status: One or more devices has experienced an unrecoverable error.  An                                                          
        attempt was made to correct the error.  Applications are unaffected.                                                      
action: Determine if the device needs to be replaced, and clear the errors                                                        
        using 'zpool clear' or replace the device with 'zpool replace'.                                                          
   see: http://illumos.org/msg/ZFS-8000-9P                                                                                        
  scan: scrub repaired 0 in 6h44m with 0 errors on Sun Apr 26 06:44:53 2015                                                      
config:                                                                                                                          
                                                                                                                                 
        NAME                                            STATE     READ WRITE CKSUM                                                
        Z1                                              ONLINE       0     0     0                                                
          raidz1-0                                      ONLINE       0     0     0                                                
            gptid/3fadc37c-89f4-11e4-afcd-d050990db2f5  ONLINE       0     0     0                                                
            gptid/40435929-89f4-11e4-afcd-d050990db2f5  ONLINE       0     0     4                                                
            gptid/40e5d232-89f4-11e4-afcd-d050990db2f5  ONLINE       0     0     0                                                
            gptid/4185c746-89f4-11e4-afcd-d050990db2f5  ONLINE       0     0     0                                                
            gptid/420f1f9d-89f4-11e4-afcd-d050990db2f5  ONLINE       0     0     0                                                
            gptid/4267b50b-89f4-11e4-afcd-d050990db2f5  ONLINE       0     0     0                                                
                                                                                                                                 
errors: No known data errors                                                                                                      
                                                                                                                                 
  pool: freenas-boot                                                                                                              
state: ONLINE                                                                                                                    
  scan: scrub repaired 0 in 0h2m with 0 errors on Sun Apr  5 03:47:25 2015                                                        
config:                                                                                                                          
                                                                                                                                 
        NAME        STATE     READ WRITE CKSUM                                                                                    
        freenas-boot  ONLINE       0     0     0                                                                                  
          da0p2     ONLINE       0     0     0                                                                                    
                                                                                                                                 
errors: No known data errors  

and:
Smartctl:

Error 2 occurred at disk power-on lifetime: 17029 hours (709 days + 13 hours)                                                    
  When the command that caused the error occurred, the device was active or idle.                                                
                                                                                                                                 
  After command completion occurred, registers were:                                                                              
  ER ST SC SN CL CH DH                                                                                                            
  -- -- -- -- -- -- --                                                                                                            
  04 71 03 83 04 11 40                                                                                                            
                                                                                                                                 
  Commands leading to the command that caused the error were:                                                                    
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name                                                                 
  -- -- -- -- -- -- -- --  ----------------  --------------------                                                                
  ea 00 00 00 00 00 40 00      00:04:01.993  FLUSH CACHE EXT                                                                      
  61 00 30 ff ff ff 4f 00      00:04:01.993  WRITE FPDMA QUEUED                                                                  
  ea 00 00 00 00 00 40 00      00:03:52.198  FLUSH CACHE EXT                                                                      
  61 00 08 ff ff ff 4f 00      00:03:52.198  WRITE FPDMA QUEUED                                                                  
  61 00 08 ff ff ff 4f 00      00:03:52.198  WRITE FPDMA QUEUED                                                                  
                                                                                                                                 
Error 1 occurred at disk power-on lifetime: 17028 hours (709 days + 12 hours)                                                    
  When the command that caused the error occurred, the device was active or idle.                                                
                                                                                                                                 
  After command completion occurred, registers were:                                                                              
  ER ST SC SN CL CH DH                                                                                                            
  -- -- -- -- -- -- --                                                                                                            
  04 71 03 83 04 11 40                                                                                                            
                                                                                                                                 
  Commands leading to the command that caused the error were:                                                                    
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name                                                                 
  -- -- -- -- -- -- -- --  ----------------  --------------------                                                                
  ea 00 00 00 00 00 40 00  32d+07:55:31.271  FLUSH CACHE EXT                                                                      
  61 00 08 d0 b5 dc 41 00  32d+07:55:31.271  WRITE FPDMA QUEUED                                                                  
  61 00 10 b8 b5 dc 41 00  32d+07:55:31.271  WRITE FPDMA QUEUED                                                                  
  61 00 08 ff ff ff 4f 00  32d+07:55:31.271  WRITE FPDMA QUEUED                                                                  
  61 00 10 ff ff ff 4f 00  32d+07:55:31.271  WRITE FPDMA QUEUED                                                                  
                                                                                                                                 
SMART Self-test log structure revision number 1                                                                                  
No self-tests have been logged.  [To run self-tests, use: smartctl -t]                                                            
                                                                                                                                 
SMART Selective self-test log data structure revision number 1                                                                    
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS                                                                                      
    1        0        0  Not_testing                                                                                              
    2        0        0  Not_testing                                                                                              
    3        0        0  Not_testing                                                                                              
    4        0        0  Not_testing                                                                                              
    5        0        0  Not_testing                                                                                              
Selective self-test flags (0x0):                                                                                                  
  After scanning selected spans, do NOT read-scan remainder of disk.                                                              
If Selective self-test is pending on power-up, resume after 0 minute delay.
"

I think there is more text on the smartctl command but this scrolls off the shell window.

Am trying to establish a telnet session....

I just freed 2 more WD reds of 3 TB and want to replace the three Seagates... one by one.. Will start to look for the proper procedure.

I believe my data was not (yet) affected, correct?

Thanks for the fast response BTW!

 

[pirateghost]

Please use the code tags available in order to format your wall of text.

 

[alexg]

This doesn't look like full smartctl -a and it seems that you have not been running smart tests periodically.

 

[nasnice]

Hi alexg,
Correct like I wrote the txt scrolls off the shell window.
Nope I have not been running smart test, my idea lesser load lesser problems....
Just how can I direct the total output of the smartctl -a /dev/ada1 to a text file on a win machine?
TIA

 

[pirateghost]

Hi alexg,
Correct like I wrote the txt scrolls of the shell window.
Nope I have not been running smart test, my idea lesser load lesser problems....
Just how can I direct the total output of the smartctl -a /dev/ada1 to a text file on a win machine?
TIA

Use PuTTY, connect, run command, and copy the output to a txt file

 

[Robert Trevellyan]

I just freed 2 more WD reds of 3 TB and want to replace the three Seagates... one by one.. Will start to look for the proper procedure.

http://doc.freenas.org/9.3/freenas_storage.html?highlight=ntfs#replacing-a-failed-drive

I believe my data was not (yet) affected, correct?

Hopefully not, but you're in a precarious situation using RAIDZ1 with disks larger than 1TB. Follow @Bidule0hm 's advice.

 

[nasnice]

Thanks,

Here is the correct smartctl output:

Code:

[root@Nas] ~# smartctl -a /dev/ada1
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p8 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST3000DM001-9YN166
Serial Number:    Z1F16SA3
LU WWN Device Id: 5 000c50 04e53d348
Firmware Version: CC4H
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun May  3 18:58:39 2015 WEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  584) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off supp                                                                                                                     ort.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 338) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_                                                                                                                     FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   117   099   006    Pre-fail  Always       -                                                                                                                            125010104
  3 Spin_Up_Time            0x0003   093   092   000    Pre-fail  Always       -                                                                                                                            0
  4 Start_Stop_Count        0x0032   099   099   020    Old_age   Always       -                                                                                                                            1930
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -                                                                                                                            1032
  7 Seek_Error_Rate         0x000f   076   060   030    Pre-fail  Always       -                                                                                                                            47934846
  9 Power_On_Hours          0x0032   081   081   000    Old_age   Always       -                                                                                                                            17033
10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -                                                                                                                            0
12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -                                                                                                                            102
183 Runtime_Bad_Block       0x0032   099   099   000    Old_age   Always       -                                                                                                                            1
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -                                                                                                                            0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -                                                                                                                            0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -                                                                                                                            4 4 4
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -                                                                                                                            0
190 Airflow_Temperature_Cel 0x0022   066   043   045    Old_age   Always   In_th                                                                                                                     e_past 34 (0 227 35 32 0)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -                                                                                                                            0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -                                                                                                                            31
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -                                                                                                                            355598
194 Temperature_Celsius     0x0022   034   057   000    Old_age   Always       -                                                                                                                            34 (0 20 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -                                                                                                                            56
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -                                                                                                                            56
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -                                                                                                                            0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -                                                                                                                            10865h+19m+22.230s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -                                                                                                                            12926762899045
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -                                                                                                                            41749799321245

SMART Error Log Version: 1
ATA Error Count: 2
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 2 occurred at disk power-on lifetime: 17029 hours (709 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle                                                                                                                     .

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 71 03 83 04 11 40

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ea 00 00 00 00 00 40 00      00:04:01.993  FLUSH CACHE EXT
  61 00 30 ff ff ff 4f 00      00:04:01.993  WRITE FPDMA QUEUED
  ea 00 00 00 00 00 40 00      00:03:52.198  FLUSH CACHE EXT
  61 00 08 ff ff ff 4f 00      00:03:52.198  WRITE FPDMA QUEUED
  61 00 08 ff ff ff 4f 00      00:03:52.198  WRITE FPDMA QUEUED

Error 1 occurred at disk power-on lifetime: 17028 hours (709 days + 12 hours)
  When the command that caused the error occurred, the device was active or idle                                                                                                                     .

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 71 03 83 04 11 40

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ea 00 00 00 00 00 40 00  32d+07:55:31.271  FLUSH CACHE EXT
  61 00 08 d0 b5 dc 41 00  32d+07:55:31.271  WRITE FPDMA QUEUED
  61 00 10 b8 b5 dc 41 00  32d+07:55:31.271  WRITE FPDMA QUEUED
  61 00 08 ff ff ff 4f 00  32d+07:55:31.271  WRITE FPDMA QUEUED
  61 00 10 ff ff ff 4f 00  32d+07:55:31.271  WRITE FPDMA QUEUED

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

This confirms that the data is not corrupt yet, correct?

 

[joeschmuck]

Yea, we need to see the ID data from the smartctl output to determine if it's a drive failure however with the specific failure message you received "Device: /dev/ada1, 56 Currently unreadable (pending) sectors (changed +8)" tends to indicate the drive is failing.

You should run a Short SMART test daily and a Long SMART Test weekly, in my opinion on all your drives.

EDIT: Yes, your drive is definitely failing. Backup your important data and then replace the failing drive. Ensure you power off your system and match the failed drive serial number to the one you removed. SATA Port 1 does not equal ada1.

 

[Robert Trevellyan]

This confirms that the data is not corrupt yet, correct?

The horizontal spacing makes this very hard to read but ... holy sh*t, 1032 reallocated sectors? Unless I'm misreading, that disk was toast a long time ago. And your Current_Pending_Sector and Offline_Uncorrectable are both 56, but even more importantly, 56 and rising.

Nothing in the SMART output confirms or denies whether any data has been lost. Either ZFS has been able to handle the situation or not, and a scrub would confirm.

But like I said, follow @Bidule0hm 's advice (and @joeschmuck 's), because with such large disks in RAIDZ1 you are on the edge.

 

[Robert Trevellyan]

MISTAKE

 

[Bidule0hm]

Nothing in the SMART output confirms or denies whether any data has been lost. Either ZFS has been able to handle the situation or not, and a scrub would confirm.

The pool status says it's ok (for now...), no data was corrupted ;)

 

[nasnice]

OK THANKS all

Going to replace the drive but...

I am trying to follow:
http://doc.freenas.org/9.3/freenas_storage.html?highlight=ntfs#replacing-a-failed-drive

There it mentions "Volume Status" Icon. I cannot find such an icon...

Because Freenas worked "out of the box" and has good documentation I never looked twice, however.... Not only did I NOT put in SMART tests but also no replication task or snapshot tasks...

What is the best way to go forward .... If I follow the link above it mentions offline buttons for individuaal disks... cant find those either....

Bottom line, just power down, locate and replace the failing disk and then use the replace disk procedure?

TIA

 

[Robert Trevellyan]

The pool status says it's ok (for now...), no data was corrupted ;)

Yes, but that only reports on data it has read. A scrub would confirm whether there's any as-yet undetected data loss.

 

[Robert Trevellyan]

I am trying to follow:
http://doc.freenas.org/9.3/freenas_storage.html?highlight=ntfs#replacing-a-failed-drive

There it mentions "Volume Status" Icon. I cannot find such an icon...

Bottom line, just power down, locate and replace the failing disk and then use the replace disk procedure?

Selecting the volume in the Storage tab will reveal the Volume Status button, just like it says in the directions:

Before physically removing the failed device, go to Storage ‣ Volumes ‣ View Volumes. Next, select your volume’s name.

Step 1 is to offline the disk, not to power off the system. Take your time and follow the directions carefully. If you don't immediately see what is described, don't ignore it assuming it's wrong.

 

[nasnice]

Found the Buttons thanks, needs getting used to the formatting of Freenas pages

Anyway I really would like to run a scrub before I offline the disk. Once again I cannot find a way to start a scrub NOW.. I can just schedule scrubs from the webUI but it makes no sense to wait for n days with the scrub while the drive may be dying...

Looked with Putty but also encountered a planning syntax; not the actual NOW command string... anyone?

 

[alexg]

From WebGUI,
Storage->Volume->Select Volume. On the bottom, you should see three icons - Detach, Scrub, and VolumeStatus

From CLI,
zpool scrub <pool-name>

 

[alexg]

By the way, find the post that explains how to setup periodic scrubs and smart tests. It will help you detect any issues earlier.