panic: I/O to pool 'main' appears to be hung on vdev

Richard Durso · Jun 7, 2019

I've been using FreeNAS since early 9.x releases and has always been incredibly stable. I recently upgraded to FreeNAS 11.2 and upgraded my four WD RED 4TB drives to WD 6TB drives -- one by one over course of a month, removed, replaced, re silvered, no problems until the last drive was replaced. I've had nothing but problems since.

Performance is very slow - I don't know how to quantify this other than saving files which use to be instant now takes 15 seconds to come back (small 3.5MB files). This is very low usage home server, I'm the only user on it. I've started to get random reboot messages:

Code:

freenas.localdomain had an unscheduled system reboot.
The operating system successfully came back online at Thu Jun  6 12:47:33 2

And I've started to get panic messages:

Code:

freenas.localdomain kernel log messages:
> panic: I/O to pool 'main' appears to be hung on vdev guid 2342498213395394601 at '/dev/gptid/85d67320-72b1-11e9-9ae5-002590dd15dd'.
> cpuid = 3
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe083c98d800
> vpanic() at vpanic+0x177/frame 0xfffffe083c98d860
> panic() at panic+0x43/frame 0xfffffe083c98d8c0
> vdev_deadman() at vdev_deadman+0x191/frame 0xfffffe083c98d910
> vdev_deadman() at vdev_deadman+0x41/frame 0xfffffe083c98d960
> vdev_deadman() at vdev_deadman+0x41/frame 0xfffffe083c98d9b0
> spa_deadman() at spa_deadman+0x87/frame 0xfffffe083c98d9e0
> taskqueue_run_locked() at taskqueue_run_locked+0x154/frame 0xfffffe083c98da40
> taskqueue_thread_loop() at taskqueue_thread_loop+0x98/frame 0xfffffe083c98da70
> fork_exit() at fork_exit+0x83/frame 0xfffffe083c98dab0
> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe083c98dab0
> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
> KDB: enter: panic
> Copyright (c) 1992-2018 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>     The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 11.2-STABLE #0 r325575+95cc58ca2a0(HEAD): Fri May 10 15:57:35 EDT 2019

The device in question ADA2 is also showing very slow "disk time" on the Reports Dashboards. See attached image.

I'm not sure how to troubleshoot this. My zpool status looks perfect (see attached image).

Jailer · Jun 7, 2019

Did you test your drives before putting them in service? Are you running periodic smart tests on your drives?

Post the output of smartctl -x /dev/ada2 in code tags.

Richard Durso · Jun 7, 2019

Thanks for the quick reply. Yes, I do a "dead-drive" check before using them. I hook each drive up to my linux desktop, format as EXT4, copy some large ISO images, leave it on for 24 hours, and then use GPARTED to delete all partitions before adding it to the freenas server. I do have SMART tests enabled for each drive.

These are BRAND NEW drives, I just got from Newegg last month. I don't understand why I see something like:

Code:

Error 21 [20] occurred at disk power-on lifetime: 14806 hours (616 days + 22 hours)

Here is full output:

Code:

root@freenas:~ # smartctl -x /dev/ada2
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD60EFRX-68MYMN1
Serial Number:    WD-WX91D65355EZ
LU WWN Device Id: 5 0014ee 2b73b3161
Firmware Version: 82.00A82
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5700 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Jun  7 08:27:13 2019 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                ( 4064) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 694) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x303d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    22
  3 Spin_Up_Time            POS--K   201   201   021    -    8933
  4 Start_Stop_Count        -O--CK   100   100   000    -    8
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   079   079   000    -    15356
 10 Spin_Retry_Count        -O--CK   100   253   000    -    0
 11 Calibration_Retry_Count -O--CK   100   253   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    8
192 Power-Off_Retract_Count -O--CK   200   200   000    -    6
193 Load_Cycle_Count        -O--CK   196   196   000    -    13193
194 Temperature_Celsius     -O---K   123   107   000    -    29
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   100   253   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   100   253   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb6  GPL,SL  VS       1  Device vendor specific log
0xb7       GPL,SL  VS      40  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 21
        CR     = Command Register
        FEATR  = Features Register
        COUNT  = Count (was: Sector Count) Register
        LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
        LH     = LBA High (was: Cylinder High) Register    ]   LBA
        LM     = LBA Mid (was: Cylinder Low) Register      ] Register
        LL     = LBA Low (was: Sector Number) Register     ]
        DV     = Device (was: Device/Head) Register
        DC     = Device Control Register
        ER     = Error register
        ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 21 [20] occurred at disk power-on lifetime: 14806 hours (616 days + 22 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 02 a2 10 0e e8 40 00  Error: UNC at LBA = 0x2a2100ee8 = 11308895976

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 08 00 58 00 02 a2 10 0e e8 40 08  5d+11:57:26.561  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 40 08  5d+11:57:26.560  READ LOG EXT
  60 00 08 00 48 00 02 a2 10 0e e8 40 08  5d+11:57:22.471  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 40 08  5d+11:57:22.471  READ LOG EXT
  60 00 08 00 38 00 02 a2 10 0e e8 40 08  5d+11:57:18.413  READ FPDMA QUEUED

Error 20 [19] occurred at disk power-on lifetime: 14806 hours (616 days + 22 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 02 a2 10 0e e8 40 00  Error: UNC at LBA = 0x2a2100ee8 = 11308895976

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 08 00 48 00 02 a2 10 0e e8 40 08  5d+11:57:22.471  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 40 08  5d+11:57:22.471  READ LOG EXT
  60 00 08 00 38 00 02 a2 10 0e e8 40 08  5d+11:57:18.413  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 40 08  5d+11:57:18.413  READ LOG EXT
  60 00 08 00 28 00 02 a2 10 0e e8 40 08  5d+11:57:14.354  READ FPDMA QUEUED

Error 19 [18] occurred at disk power-on lifetime: 14806 hours (616 days + 22 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 02 a2 10 0e e8 40 00  Error: UNC at LBA = 0x2a2100ee8 = 11308895976

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 08 00 38 00 02 a2 10 0e e8 40 08  5d+11:57:18.413  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 40 08  5d+11:57:18.413  READ LOG EXT
  60 00 08 00 28 00 02 a2 10 0e e8 40 08  5d+11:57:14.354  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 40 08  5d+11:57:14.354  READ LOG EXT
  60 00 08 00 18 00 02 a2 10 0e e8 40 08  5d+11:57:10.270  READ FPDMA QUEUED

Error 18 [17] occurred at disk power-on lifetime: 14806 hours (616 days + 22 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 02 a2 10 0e e8 40 00  Error: UNC at LBA = 0x2a2100ee8 = 11308895976

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 08 00 28 00 02 a2 10 0e e8 40 08  5d+11:57:14.354  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 40 08  5d+11:57:14.354  READ LOG EXT
  60 00 08 00 18 00 02 a2 10 0e e8 40 08  5d+11:57:10.270  READ FPDMA QUEUED
  61 00 10 00 10 00 00 00 40 02 90 40 08  5d+11:57:10.270  WRITE FPDMA QUEUED
  61 00 10 00 08 00 02 ba a0 f2 90 40 08  5d+11:57:10.163  WRITE FPDMA QUEUED

Error 17 [16] occurred at disk power-on lifetime: 14806 hours (616 days + 22 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 02 a2 10 0e e8 40 00  Error: UNC at LBA = 0x2a2100ee8 = 11308895976

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 08 00 18 00 02 a2 10 0e e8 40 08  5d+11:57:10.270  READ FPDMA QUEUED
  61 00 10 00 10 00 00 00 40 02 90 40 08  5d+11:57:10.270  WRITE FPDMA QUEUED
  61 00 10 00 08 00 02 ba a0 f2 90 40 08  5d+11:57:10.163  WRITE FPDMA QUEUED
  61 00 10 00 00 00 02 ba a0 f0 90 40 08  5d+11:57:10.159  WRITE FPDMA QUEUED
  60 00 28 00 f8 00 02 a0 7e 10 b8 40 08  5d+11:57:10.080  READ FPDMA QUEUED

Error 16 [15] occurred at disk power-on lifetime: 14806 hours (616 days + 22 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 02 a2 10 0e e8 40 00  Error: UNC at LBA = 0x2a2100ee8 = 11308895976

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 28 00 90 00 02 a2 23 b3 a8 40 08  5d+11:57:05.946  READ FPDMA QUEUED
  60 00 80 00 88 00 02 7d cc ab 70 40 08  5d+11:57:05.946  READ FPDMA QUEUED
  60 00 08 00 80 00 02 a2 10 0e e8 40 08  5d+11:57:05.946  READ FPDMA QUEUED
  60 00 28 00 78 00 02 a0 7e 10 b8 40 08  5d+11:57:05.946  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 40 08  5d+11:57:05.945  READ LOG EXT

Error 15 [14] occurred at disk power-on lifetime: 14806 hours (616 days + 22 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 02 a2 10 0e e8 40 00  Error: UNC at LBA = 0x2a2100ee8 = 11308895976

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 28 00 68 00 02 a2 23 b3 a8 40 08  5d+11:57:01.877  READ FPDMA QUEUED
  60 00 80 00 60 00 02 7d cc ab 70 40 08  5d+11:57:01.877  READ FPDMA QUEUED
  60 00 08 00 58 00 02 a2 10 0e e8 40 08  5d+11:57:01.877  READ FPDMA QUEUED
  60 00 28 00 50 00 02 a0 7e 10 b8 40 08  5d+11:57:01.877  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 40 08  5d+11:57:01.876  READ LOG EXT

Error 14 [13] occurred at disk power-on lifetime: 14806 hours (616 days + 22 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 02 a2 10 0e e8 40 00  Error: UNC at LBA = 0x2a2100ee8 = 11308895976

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 08 00 40 00 02 52 14 2e f0 40 08  5d+11:56:57.621  READ FPDMA QUEUED
  60 00 08 00 38 00 01 e4 40 30 58 40 08  5d+11:56:57.621  READ FPDMA QUEUED
  60 00 80 00 30 00 01 a4 4b bf 68 40 08  5d+11:56:57.621  READ FPDMA QUEUED
  60 00 08 00 28 00 00 7c 59 54 b0 40 08  5d+11:56:57.621  READ FPDMA QUEUED
  60 00 08 00 20 00 02 a2 0d 95 78 40 08  5d+11:56:57.621  READ FPDMA QUEUED

SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    29 Celsius
Power Cycle Min/Max Temperature:     23/29 Celsius
Lifetime    Min/Max Temperature:      2/45 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (400)

Index    Estimated Time   Temperature Celsius
 401    2019-06-07 00:30    29  **********
 ...    ..(142 skipped).    ..  **********
  66    2019-06-07 02:53    29  **********
  67    2019-06-07 02:54    28  *********
 ...    ..(155 skipped).    ..  *********
 223    2019-06-07 05:30    28  *********
 224    2019-06-07 05:31    29  **********
 ...    ..(175 skipped).    ..  **********
 400    2019-06-07 08:27    29  **********

SCT Error Recovery Control:
           Read:     70 (7.0 seconds)
          Write:     70 (7.0 seconds)

Device Statistics (GP/SMART Log 0x04) not supported

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2           13  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           15  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4       658184  Vendor specific

danb35 · Jun 7, 2019

Richard Durso said:
Yes, I do a "dead-drive" check before using them.

So, no, you don't test them. See Uncle Fester's Guide (link in my sig) for details on how to test.

Richard Durso said:
9 Power_On_Hours -O--CK 079 079 000 - 15356

This is not a brand-new drive; it's been powered on for nearly two years. Either you mis-read the listing, or Newegg messed up royally.

Richard Durso said:
No self-tests have been logged. [To run self-tests, use: smartctl -t]

...and you've never run a SMART self-test on it. Run one. smartctl -t long /dev/ada2.

Richard Durso · Jun 7, 2019

This is not a brand-new drive; it's been powered on for nearly two years. Either you mis-read the listing, or Newegg messed up royally.

I'll be calling them ASAP. This was sold as new in sealed anti static bag.

...and you've never run a SMART self-test on it. Run one. smartctl -t long /dev/ada2.

Clarification question on the "never" ... under Storage > Disks > Enable SMART is set to yes for all devices. Under Services > SMART is enabled, Start Automatically is checked. Did I miss something for the shorts test run automatically?

Anyway, thanks for the suggestion. Running now.

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 694 minutes for test to complete.
Test will complete after Fri Jun 7 21:33:36 2019

danb35 · Jun 7, 2019

Richard Durso said:
Did I miss something for the shorts test run automatically?

Yes, you did--you need to actually schedule the tests. https://www.familybrown.org/dokuwiki/doku.php?id=fester112:smart_schedule

Richard Durso · Jun 7, 2019

danb35 said:
Yes, you did--you need to actually schedule the tests. https://www.familybrown.org/dokuwiki/doku.php?id=fester112:smart_schedule

OK, that is already set as well. It's set to run every 12 hours, every day. See attached.

Richard Durso · Jun 7, 2019

I've changed it to "Daily ( 0 0 * * *)" instead of the custom one it currently had.

danb35 · Jun 7, 2019

Richard Durso said:
OK, that is already set as well.

The screen shot you posted doesn't have any disks selected. You need to select every disk you want the test to run on.

Richard Durso · Jun 7, 2019

danb35 said:
The screen shot you posted doesn't have any disks selected. You need to select every disk you want the test to run on.

So... "*" does not mean all-disks? Darn. I've now set drives explicitly.

Chris Moore · Jun 7, 2019

Richard Durso said:
These are BRAND NEW drives, I just got from Newegg last month. I don't understand why I see something like:

Your "dead drive check" is not very comprehensive. There are many drive faults that don't show up until the drive has been running for days or even weeks

rvassar · Jun 7, 2019

Richard Durso said:
Code:
9 Power_On_Hours -O--CK 079 079 000 - 15356

They sold that as new?!? I'd say you got a refurb. Did it come in a sealed WD branded box? Or was it a bare drive in a heat sealed static bag?

Chris Moore · Jun 7, 2019

It could be like this one:
https://www.newegg.com/p/1A0-00F3-000F6

They have a lot of items on NewEgg that are not actually new.

Richard Durso · Jun 7, 2019

Chris Moore said:
Your "dead drive check" is not very comprehensive. There are many drive faults that don't show up until the drive has been running for days or even weeks

No argument on that. I know, it's just a home server not doing much. This is not a business or production environment. I've made notes for next time to test drive better before use.

Richard Durso · Jun 7, 2019

rvassar said:
They sold that as new?!? I'd say you got a refurb. Did it come in a sealed WD branded box? Or was it a bare drive in a heat sealed static bag?

I called Newegg and they confirmed it was not supposed to be used or refurb. I'm getting a RMA. It did arrive in sealed static bag. It did NOT arrive in the typical WD cardboard box as my previous drives did.

Richard Durso · Jun 8, 2019

Yesterday when I started the long test, it reported:

# smartctl -t long /dev/ada2
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 694 minutes for test to complete.
Test will complete after Fri Jun 7 21:33:36 2019

Should have completed last night. This morning still says no self-tests have been logged...

# smartctl -l selftest /dev/ada2
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

Suggestion?

Jailer · Jun 8, 2019

Run smartctl -x /dev/ada2 again to see if it's still running.

Richard Durso · Jun 8, 2019

Jailer said:
Run smartctl -x /dev/ada2 again to see if it's still running.

The "-x" didn't show anything about an existing test running. However, trying to run a "-t long" again did report this:

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Can't start self-test without aborting current test (10% remaining),
add '-t force' option to override, or run 'smartctl -X' to abort test.

I assume that is what you meant. Thanks. I'll check again in a few hours.

Richard Durso · Jun 9, 2019

So I waited another day and it's still at 10% remaining...

# smartctl -t long /dev/ada2
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Can't start self-test without aborting current test (10% remaining),
add '-t force' option to override, or run 'smartctl -X' to abort test.

Jailer · Jun 9, 2019

And now you see why you were getting the error message you were. You've discovered the drive that is causing the vdev to "hang". Get it replaced.

Important Announcement for the TrueNAS Community.

panic: I/O to pool 'main' appears to be hung on vdev

Explorer

Attachments

Not strong, but bad

Explorer

Hall of Famer

Explorer

Hall of Famer

Explorer

Attachments

Explorer

Hall of Famer

Explorer

Attachments

Hall of Famer

Guru

Hall of Famer

Explorer

Explorer

Explorer

Not strong, but bad

Explorer

Explorer

Not strong, but bad

Similar threads