Degraded, drive shows "removed" but I didn't remove it

[sremick]

So maybe this is simple. Maybe FreeNAS just glitched. Maybe all I have to do is add the drive back. But I don't want to fsck this up so I'm checking here first before I do anything.
Version: FreeNAS-11.3-U4.1

Single RAIDZ2 vdev, 6 disks. But ada2 is now showing as "removed" even though I didn't remove it. Alert says: "Pool MainVolume state is DEGRADED: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state.."

It seems FreeNAS still sees the drive as being there, so its removal/absence from the vdev is just a software thing...? From the dashboard I can click into it, get its GUID, see that there are no error counts on anything. Under Storage -> Disks I also see the drive, but under "Pool" it says "unused". But data under serial#, model, size, type, etc.

Should I just try to "Online" the drive? I want to be careful here... I'm not sure how the drive got "removed" but it's almost as if there's no error on it otherwise.

 

[Jailer]

If it was kicked from your pool there is an issue with the drive. I'd start by checking out the smart data on the drive.

 

[sremick]

Had to reboot for another reason, and now the drive is back online. But here's the SMART data:

Code:

# smartctl -a /dev/ada2
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p11 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD80EFAX-68LHPN0
Serial Number:    xxxxxxxxxxxx
LU WWN Device Id: 5 000cca 257e4a7e5
Firmware Version: 83.H0A83
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Sep 12 14:04:05 2020 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (   93) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      (1036) minutes.
SCT capabilities:            (0x003d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0004   127   127   054    Old_age   Offline      -       120
  3 Spin_Up_Time            0x0007   253   253   024    Pre-fail  Always       -       219 (Average 264)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       122
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0004   128   128   020    Old_age   Offline      -       18
  9 Power_On_Hours          0x0012   098   098   000    Old_age   Always       -       14783
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       120
 22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       697
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       697
194 Temperature_Celsius     0x0002   171   171   000    Old_age   Always       -       38 (Min/Max 24/53)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     14772         -
# 2  Short offline       Completed without error       00%     14364         -
# 3  Short offline       Completed without error       00%     14200         -
# 4  Short offline       Completed without error       00%     14032         -
# 5  Short offline       Completed without error       00%     13878         -
# 6  Short offline       Completed without error       00%     13638         -
# 7  Extended offline    Completed without error       00%     13570         -
# 8  Short offline       Completed without error       00%     13470         -
# 9  Short offline       Completed without error       00%     13302         -
#10  Extended offline    Completed without error       00%     13232         -
#11  Short offline       Completed without error       00%     13134         -
#12  Short offline       Completed without error       00%     12918         -
#13  Extended offline    Completed without error       00%     12846         -
#14  Short offline       Completed without error       00%     12750         -
#15  Short offline       Completed without error       00%     12582         -
#16  Extended offline    Completed without error       00%     12508         -
#17  Short offline       Completed without error       00%     12414         -
#18  Short offline       Completed without error       00%     12174         -
#19  Extended offline    Completed without error       00%     12099         -
#20  Short offline       Completed without error       00%     12006         -
#21  Short offline       Completed without error       00%     11838         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[Apollo]

Had to reboot for another reason, and now the drive is back online. But here's the SMART data:

Code:

# smartctl -a /dev/ada2
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p11 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD80EFAX-68LHPN0
Serial Number:    xxxxxxxxxxxx
LU WWN Device Id: 5 000cca 257e4a7e5
Firmware Version: 83.H0A83
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Sep 12 14:04:05 2020 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (   93) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      (1036) minutes.
SCT capabilities:            (0x003d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0004   127   127   054    Old_age   Offline      -       120
  3 Spin_Up_Time            0x0007   253   253   024    Pre-fail  Always       -       219 (Average 264)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       122
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0004   128   128   020    Old_age   Offline      -       18
  9 Power_On_Hours          0x0012   098   098   000    Old_age   Always       -       14783
10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       120
22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       697
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       697
194 Temperature_Celsius     0x0002   171   171   000    Old_age   Always       -       38 (Min/Max 24/53)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     14772         -
# 2  Short offline       Completed without error       00%     14364         -
# 3  Short offline       Completed without error       00%     14200         -
# 4  Short offline       Completed without error       00%     14032         -
# 5  Short offline       Completed without error       00%     13878         -
# 6  Short offline       Completed without error       00%     13638         -
# 7  Extended offline    Completed without error       00%     13570         -
# 8  Short offline       Completed without error       00%     13470         -
# 9  Short offline       Completed without error       00%     13302         -
#10  Extended offline    Completed without error       00%     13232         -
#11  Short offline       Completed without error       00%     13134         -
#12  Short offline       Completed without error       00%     12918         -
#13  Extended offline    Completed without error       00%     12846         -
#14  Short offline       Completed without error       00%     12750         -
#15  Short offline       Completed without error       00%     12582         -
#16  Extended offline    Completed without error       00%     12508         -
#17  Short offline       Completed without error       00%     12414         -
#18  Short offline       Completed without error       00%     12174         -
#19  Extended offline    Completed without error       00%     12099         -
#20  Short offline       Completed without error       00%     12006         -
#21  Short offline       Completed without error       00%     11838         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Looks like a Shingle Magnetic Drive.

 

[joeschmuck]

Looks like a Shingle Magnetic Drive.

I don't think so, the 8TB drives are CMR, unless that too has changed.

I don't see anything obviously wrong with your SMART data. Keep an eye on it, if your drive gets tossed again, shut down your system and swap the SATA cable between the suspect drive and another drive. If the failure follows to the other drive then replace the data cable, if the problem happens again to the same drive then reseat the cables (including power). If it happens again to the same drive and you have moved it to another physical location, then the drive is likely bad. Odds are you have a cable issue, well let's hope. Also you could run another SMART Extended/Long test and see if anything changes in the ID values 5, 196, 197, 199 and if it completes a long test of course.

Good Luck.

 

[Jailer]

If you haven't already I would check the smart data on all of your drives just to rule out the possibility that you took the smart data from the wrong drive.

 

[joeschmuck]

Also, track your drives by serial number, not ada0, ada1, etc...

 

[Apollo]

I don't think so, the 8TB drives are CMR, unless that too has changed.

I don't see anything obviously wrong with your SMART data. Keep an eye on it, if your drive gets tossed again, shut down your system and swap the SATA cable between the suspect drive and another drive. If the failure follows to the other drive then replace the data cable, if the problem happens again to the same drive then reseat the cables (including power). If it happens again to the same drive and you have moved it to another physical location, then the drive is likely bad. Odds are you have a cable issue, well let's hope. Also you could run another SMART Extended/Long test and see if anything changes in the ID values 5, 196, 197, 199 and if it completes a long test of course.

Good Luck.

I think this is the all EFAX fuss about.

Just to be on the safe side, I checked my info from the following source which states the technology of the drive :

WD Red NAS Hard Drive, Internal Hard Drive (2 TB to 6 TB) | Western Digital | Western Digital

There’s a leading edge WD Red™ drive for every compatible NAS system to store your precious data. With drives up to 6TB, WD Red™ offers a wide array of storage for customers looking to build a NAS solution. Built and tested for personal and home NAS systems with up to 8-bays, these drives pack...

shop.westerndigital.com

• designed to thrive in multi-bay NAS system environments.

SMR
WD20EFAX (2TB)
WD30EFAX (3TB)
WD40EFAX (4TB)
WD60EFAX (6TB)

I must conceed they are not refering to the 8TB model, but then I beg the question about the EFAX model number.
From my point of view the EFAX is the SMR version of the EFRX CMR version, unless my reasoning skill is betraying me.

Just my common sense.

Just digging deeper:

Clearance - WD Red Pro NAS Hard Drive | Western Digital Store

For up to 24 bay medium to large-scale NAS environments, the WD Red Pro drive is engineered to handle increased business workloads and carries a 5-year limited warranty.

www.westerndigital.com

So who should I trust? Only if I knew what is the naming scheme used by WD.

Complete nonsense to me.

 

[joeschmuck]

So who should I trust?

Well you can trust no one really but cross your fingers and hope WD is telling the truth. Right now your drive is considered a CMR drive until someone proves otherwise.

Complete nonsense to me.

To us as well. We had a great relationship with WD and then they started putting SMR drives into the NAS lineup. They took some heat for it and I'm sure they lost many purchase orders as well. Reputation is everything in this business.

So keep plucking along and hopefully your drive dropping offline will never occur again.

 

[sremick]

Thanks for your advice. I'll definitely be keeping an eye on it and I do track by serial #. :) I too am hoping it's just a cable issue and that was one of my first thoughts.

I too was under the impression that the 8TB weren't affected by the SMR issue (which I followed very closely).