SOLVED Need help please-FN 11.2.U5 Unusable Pool-ports using Sata Express (backward to Sata only ports)

Joined
Oct 18, 2018
Messages
969
199 UDMA_CRC_Error_Count 0x003e 200 192 000 Old_age Always - 178
The fact that that last number is not a 0 indicates an issue communicating with the system. Most often this is a cable issue, it may have become lose, but other times it could be a failing controller either on the motherboard or add on card. These could certainly put your pool in an unknown state. I'm curious to see the results for all of your drives once the tests complete.
 

1kokies

Contributor
Joined
Oct 7, 2017
Messages
138
The fact that that last number is not a 0 indicates an issue communicating with the system. Most often this is a cable issue, it may have become lose, but other times it could be a failing controller either on the motherboard or add on card. These could certainly put your pool in an unknown state. I'm curious to see the results for all of your drives once the tests complete.

this interpretation is very helpful even at this point, thanks. This disk and all other in this pool, (should have separated some to mobo SATA) is plugged to the raid card and the cable is this https://www.amazon.com/Cable-Matters-Internal-SFF-8087-Breakout/dp/B012BPLYJC. the first disk scanned is at 30%, while the rest is at 80-90%. should be completed within 10 hours. will update and post complete results
 

1kokies

Contributor
Joined
Oct 7, 2017
Messages
138
current controller arrangement are 8 disks in affected pool are plugged to raid card, previously before pool unknown was
1. Data Storage (affected pool) 4 disk plugged to raid card
2. Data Media 4 disk plugged to raid card
3. Data Storage (affected pool) 4 disk plugged to mobo
4. Slog 1 disk plugged to mobo
 

1kokies

Contributor
Joined
Oct 7, 2017
Messages
138
all smart scan done for da0-da3, attached results (post max 3k characters). da4-da7 in next post.
Comments and opinions much appreciated

The disks & pool looks ok, if cables or controller is the suspected problem , will use new ones and isolate one by one

Code:
root@freenas:~ # smartctl -a /dev/da0
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST4000VN008-2DR166
Serial Number:    ZDH1E7MA
LU WWN Device Id: 5 000c50 0a2ca53df
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Oct 18 19:06:00 2019 +08
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  581) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 631) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   082   064   044    Pre-fail  Always       -       176119726
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       540
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   090   060   045    Pre-fail  Always       -       906516910
  9 Power_On_Hours          0x0032   081   081   000    Old_age   Always       -       16872 (64 249 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       545
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   001   000    Old_age   Always       -       667
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   060   055   040    Old_age   Always       -       40 (Min/Max 33/45)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       205
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       891
194 Temperature_Celsius     0x0022   040   045   000    Old_age   Always       -       40 (0 21 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   192   000    Old_age   Always       -       178
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       16862 (237 254 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       65529703758
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       46229415874

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     16865         -
# 2  Short offline       Completed without error       00%     16857         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@freenas:~ #

root@freenas:~ # smartctl -a /dev/da1
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST4000VN008-2DR166
Serial Number:    ZGY02DMS
LU WWN Device Id: 5 000c50 0a2c9d3c3
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Oct 18 17:50:32 2019 +08
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  591) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 646) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   072   064   044    Pre-fail  Always       -       17564311
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       528
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   088   060   045    Pre-fail  Always       -       679162100
  9 Power_On_Hours          0x0032   081   081   000    Old_age   Always       -       16845 (174 175 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       532
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   064   056   040    Old_age   Always       -       36 (Min/Max 31/42)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       225
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       959
194 Temperature_Celsius     0x0022   036   044   000    Old_age   Always       -       36 (0 22 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       16838 (215 51 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       65542818809
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       46208949750

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     16843         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@freenas:~ #

root@freenas:~ # smartctl -a /dev/da2
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST4000VN008-2DR166
Serial Number:    ZGY02E9G
LU WWN Device Id: 5 000c50 0a2c98452
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Oct 18 17:56:13 2019 +08
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  581) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 616) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   075   064   044    Pre-fail  Always       -       33043800
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       526
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   089   060   045    Pre-fail  Always       -       871349013
  9 Power_On_Hours          0x0032   081   081   000    Old_age   Always       -       16850 (131 92 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       530
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   090   000    Old_age   Always       -       4295032844
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   056   046   040    Old_age   Always       -       44 (Min/Max 35/54)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       194
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       924
194 Temperature_Celsius     0x0022   044   054   000    Old_age   Always       -       44 (0 22 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       10
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       16842 (245 26 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       68351130361
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       47441722477

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     16848         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@freenas:~ #

root@freenas:~ # smartctl -a /dev/da3
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST4000VN008-2DR166
Serial Number:    ZDH1E7GD
LU WWN Device Id: 5 000c50 0a2caa646
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Oct 18 18:01:07 2019 +08
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  591) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 634) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   083   064   044    Pre-fail  Always       -       193439027
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       238
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   088   060   045    Pre-fail  Always       -       594037128
  9 Power_On_Hours          0x0032   083   083   000    Old_age   Always       -       15607 (99 240 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       239
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   059   040   040    Old_age   Always   In_the_past 41 (Min/Max 33/51)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       191
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       423
194 Temperature_Celsius     0x0022   041   060   000    Old_age   Always       -       41 (0 22 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       15592 (186 188 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       58283525612
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       29460162131

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     15604         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@freenas:~ #

 

1kokies

Contributor
Joined
Oct 7, 2017
Messages
138
smart scan for da4-da7

Code:

root@freenas:~ # smartctl -a /dev/da4
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST4000VN008-2DR166
Serial Number:    ZDH1NGK4
LU WWN Device Id: 5 000c50 0a317875a
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Oct 18 18:07:07 2019 +08
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  591) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 636) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   076   064   044    Pre-fail  Always       -       38570718
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       234
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   089   060   045    Pre-fail  Always       -       754316353
  9 Power_On_Hours          0x0032   083   083   000    Old_age   Always       -       15613 (74 180 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       235
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   063   040   040    Old_age   Always   In_the_past 37 (Min/Max 31/46)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       227
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       424
194 Temperature_Celsius     0x0022   037   060   000    Old_age   Always       -       37 (0 22 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       15592 (158 248 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       60902870916
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       29207973635

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     15611         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@freenas:~ #

root@freenas:~ # smartctl -a /dev/da5
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST4000VN008-2DR166
Serial Number:    ZDH1NGQN
LU WWN Device Id: 5 000c50 0a3178b7d
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Oct 18 18:11:46 2019 +08
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  591) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 643) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   080   064   044    Pre-fail  Always       -       98585899
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       231
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   088   060   045    Pre-fail  Always       -       654032983
  9 Power_On_Hours          0x0032   083   083   000    Old_age   Always       -       15606 (248 72 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       232
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   055   041   040    Old_age   Always       -       45 (Min/Max 35/56)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       185
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       414
194 Temperature_Celsius     0x0022   045   059   000    Old_age   Always       -       45 (0 22 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       15587 (236 31 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       60909784332
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       27064090934

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     15604         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@freenas:~ #

root@freenas:~ # smartctl -a /dev/da6
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST4000VN008-2DR166
Serial Number:    ZDH1E73Y
LU WWN Device Id: 5 000c50 0a2ca86e0
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Oct 18 18:12:40 2019 +08
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  591) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 642) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   082   064   044    Pre-fail  Always       -       171196199
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       232
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   088   060   045    Pre-fail  Always       -       592364782
  9 Power_On_Hours          0x0032   083   083   000    Old_age   Always       -       15594 (144 127 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       232
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       1
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   061   043   040    Old_age   Always       -       39 (Min/Max 32/49)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       215
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       431
194 Temperature_Celsius     0x0022   039   057   000    Old_age   Always       -       39 (0 21 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       15577 (87 218 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       58260424356
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       27312759066

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     15592         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@freenas:~ #

root@freenas:~ # smartctl -a /dev/da7
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST4000VN008-2DR166
Serial Number:    ZDH1NJ64
LU WWN Device Id: 5 000c50 0a317467f
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Oct 18 18:13:44 2019 +08
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  591) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 654) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   076   064   044    Pre-fail  Always       -       38866261
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       269
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   088   060   045    Pre-fail  Always       -       672135520
  9 Power_On_Hours          0x0032   083   083   000    Old_age   Always       -       15631 (85 4 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       270
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   060   048   040    Old_age   Always       -       40 (Min/Max 33/49)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       230
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       475
194 Temperature_Celsius     0x0022   040   052   000    Old_age   Always       -       40 (0 22 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       15611 (60 10 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       60920103562
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       31355115592

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     15629         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@freenas:~ #

 
Joined
Oct 18, 2018
Messages
969
Hmm, alright so it looks like two of your drives, at one point, had difficulty communicating with the controller. I can see this because the value for UDMA_CRC_Error_Count is > 0 for two drives. That value will typically not get reset; so it if doesn't continue to increate it is likely that while you moved your hardware around etc that it resolved itself.

Alright, I'm nearly ready to offer my idea for a solution. :) Before I do though, can you clarify whether you have the old and new SLOG SSDs inserted currently?

Also, could you give me the full output of zpool status? The copy-paste you did above was extremely helpful, could you do the same with this command? It will print info about all of your pools including, if it is recognized, the affected pool. Could you do the same with zpool import?
 

1kokies

Contributor
Joined
Oct 7, 2017
Messages
138
:) great to hear from you

'idle hands in the devil's workshop' & i have done following so far

"REPAIR LOG 181019

HDA Raid card A
port0-da0, da2, da5 & da3
port1-da1, da4, da6 & da7

1. da0 & da2 appears to have UDMA & Command error. They are both on the same HBA raid card AND on the same
mini SAS 1 port to SATA 4 port cable. Replaced the cable port 0, to power up. All 15 disks detected but Pool unknown
2. Replace cable port 1, to power up, all disks present, Pool unknown
3. Swap HBA Raid to another pcie lane, all disks present, Pool unknown
4. Replace new HBA Raid card, all disks present, Pool unknown"

Hmm, all right so it looks like two of your drives, at one point, had difficulty communicating with the controller. I can see this because the value for UDMA_CRC_Error_Count is > 0 for two drives. That value will typically not get reset; so it if doesn't continue to increate it is likely that while you moved your hardware around etc that it resolved itself.
agreed, da0 and da2 are the disks.

all right, I'm nearly ready to offer my idea for a solution. :) Before I do though, can you clarify whether you have the old and new SLOG SSDs inserted currently?
yes both are always plugged in

Also, could you give me the full output of zpool status? The copy-paste you did above was extremely helpful, could you do the same with this command? It will print info about all of your pools including, if it is recognized, the affected pool. Could you do the same with zpool import?
glad you liked the copy-paste :)

Code:

root@freenas:~ # zpool status
  pool: Data-Media
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: resilvered 56K in 0 days 00:00:02 with 0 errors on Thu Oct 17 00:17:09 2019
config:

        NAME                                            STATE     READ WRITE CKSUM
        Data-Media                                      ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/151f2837-e61d-11e7-a41a-086266295468  ONLINE       0     0     0
            gptid/bb0c2d26-e5db-11e7-b7ed-086266295468  ONLINE       0     0     0
            gptid/b70cb8cd-e4de-11e7-b58d-086266295468  ONLINE       0     0     0
            gptid/b84b6866-e4de-11e7-b58d-086266295468  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:12 with 0 errors on Wed Oct 16 03:45:12 2019
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da8p2     ONLINE       0     0     0

errors: No known data errors
root@freenas:~ #

root@freenas:~ # zpool import
   pool: Data-Storage
     id: 1993046496049455044
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-6X
 config:

        Data-Storage                                    UNAVAIL  missing device
          raidz2-0                                      ONLINE
            gptid/13105a29-c624-11e7-bbe0-086266295468  ONLINE
            gptid/1405e572-c624-11e7-bbe0-086266295468  ONLINE
            gptid/e3502be7-ec58-11e7-81f8-086266295468  ONLINE
            gptid/bfdede03-ed42-11e7-969f-086266295468  ONLINE
          raidz2-1                                      ONLINE
            gptid/367a9015-c625-11e7-bbe0-086266295468  ONLINE
            gptid/18a60acb-ecae-11e7-af1c-086266295468  ONLINE
            gptid/8cc6f7c5-ec2f-11e7-8da4-086266295468  ONLINE
            gptid/0544db8f-ece7-11e7-ad97-086266295468  ONLINE
        logs
          1100341364746838742                           UNAVAIL  corrupted data

        Additional devices are known to be part of this pool, though their
        exact configuration cannot be determined.
root@freenas:~ #
 
Joined
Oct 18, 2018
Messages
969
Hey @1kokies, thanks for the output. Very good news, I think. The reason your pool is hosed is just because of that darned log device. :) So, lets tell the system to stop using it! What happens if you do zpool remove Data-Storage 1100341364746838742? After that is the pool available and if so what is the output of zpool status Date-Storage? If not, go ahead and reboot. Is it available now? If so, what is the output of zpool status Data-Storage and if not what is the output of zpool import?

My bet is that whatever we do to solve the problem will require removing the SLOG device, which is what the above commands do. Once we get the pool imported again you'll be able to add the SLOG device back to your pool and use the good SLOG device.
 

1kokies

Contributor
Joined
Oct 7, 2017
Messages
138
Hey @1kokies, thanks for the output. Very good news, I think. The reason your pool is hosed is just because of that darned log device. :) So, lets tell the system to stop using it! What happens if you do zpool remove Data-Storage 1100341364746838742? After that is the pool available? If not, go ahead and reboot. Is it available now? If so, what is the output of zpool status Data-Storage?
:) glad we could close the gap and agree on that slog thingy. We stop using it and no need to insert it back also. it's ok if the IO speed is slower, i will manage. But i am not very familiar with shell CLI ya so need a step by step guide, hope you are ok with it.
1. do a
Code:
zpool remove Data-Storage 1100341364746838742
and see if pool is available
2. If it is not available i will re boot and see if pool is available.
3. Then do
Code:
zpool status Data-Storage


My bet is that whatever we do to solve the problem will require removing the SLOG device, which is what the above commands do. Once we get the pool imported again you'll be able to add the SLOG device back to your pool and use the good SLOG device.
let's agree to remove and not add back any slog ya :)
 
Joined
Oct 18, 2018
Messages
969
  1. zpool remove Data-Storage 1100341364746838742
  2. If the pool is available
    1. zpool status Data-Storage
  3. If the pool is not available
    1. Reboot
    2. zpool status Data-Storage
  4. If the pool is STILL not available
    1. zpool import
At any point in the above if an error is encountered stop and report back.


let's agree to remove and not add back any slog ya :)
Not necessarily. This is a long discussion though and worth having after we get your pool running again.
 

1kokies

Contributor
Joined
Oct 7, 2017
Messages
138
OK did step #1 and returned following

Code:
root@freenas:~ # zpool import
   pool: Data-Storage
     id: 1993046496049455044
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-6X
 config:

        Data-Storage                                    UNAVAIL  missing device
          raidz2-0                                      ONLINE
            gptid/13105a29-c624-11e7-bbe0-086266295468  ONLINE
            gptid/1405e572-c624-11e7-bbe0-086266295468  ONLINE
            gptid/e3502be7-ec58-11e7-81f8-086266295468  ONLINE
            gptid/bfdede03-ed42-11e7-969f-086266295468  ONLINE
          raidz2-1                                      ONLINE
            gptid/367a9015-c625-11e7-bbe0-086266295468  ONLINE
            gptid/18a60acb-ecae-11e7-af1c-086266295468  ONLINE
            gptid/8cc6f7c5-ec2f-11e7-8da4-086266295468  ONLINE
            gptid/0544db8f-ece7-11e7-ad97-086266295468  ONLINE
        logs
          1100341364746838742                           UNAVAIL  corrupted data

        Additional devices are known to be part of this pool, though their
        exact configuration cannot be determined.
root@freenas:~ #
root@freenas:~ #
root@freenas:~ #
root@freenas:~ #
root@freenas:~ # zpool remove Data-Storage 1100341364746838742
cannot open 'Data-Storage': no such pool
root@freenas:~ #
 
Joined
Oct 18, 2018
Messages
969
Alright, I expected as much. What I'd try at this point is to export your pool via the GUI, do NOT click destroy data. And then try to import it again via the GUI. Once you do that what is the output of both zpool status Data-Storage and zpool import?
 

1kokies

Contributor
Joined
Oct 7, 2017
Messages
138
all right, I expected as much. What I'd try at this point is to export your pool via the GUI, do NOT click destroy data. And then try to import it again via the GUI. Once you do that what is the output of both zpool status Data-Storage and zpool import?
okok just to be extremely sure, i click on the box 'Confirm export/disconnect' only right ? screenshot attached
 

Attachments

  • exportpool1.PNG
    exportpool1.PNG
    42 KB · Views: 232
Joined
Oct 18, 2018
Messages
969
Correct.
 

1kokies

Contributor
Joined
Oct 7, 2017
Messages
138
OK done the GUI export but when 'import an existing pool' there is no pool to import

Output zpool status Data-Storage & zpool import

Code:
root@freenas:~ # zpool status Data-Storage
cannot open 'Data-Storage': no such pool

root@freenas:~ # zpool import
   pool: Data-Storage
     id: 1993046496049455044
  state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-6X
config:

        Data-Storage                                    UNAVAIL  missing device
          raidz2-0                                      ONLINE
            gptid/13105a29-c624-11e7-bbe0-086266295468  ONLINE
            gptid/1405e572-c624-11e7-bbe0-086266295468  ONLINE
            gptid/e3502be7-ec58-11e7-81f8-086266295468  ONLINE
            gptid/bfdede03-ed42-11e7-969f-086266295468  ONLINE
          raidz2-1                                      ONLINE
            gptid/367a9015-c625-11e7-bbe0-086266295468  ONLINE
            gptid/18a60acb-ecae-11e7-af1c-086266295468  ONLINE
            gptid/8cc6f7c5-ec2f-11e7-8da4-086266295468  ONLINE
            gptid/0544db8f-ece7-11e7-ad97-086266295468  ONLINE
        logs
          1100341364746838742                           UNAVAIL  corrupted data

        Additional devices are known to be part of this pool, though their
        exact configuration cannot be determined.
root@freenas:~ #
 

Attachments

  • exportpool2.PNG
    exportpool2.PNG
    49.9 KB · Views: 215

1kokies

Contributor
Joined
Oct 7, 2017
Messages
138
i have not re booted though
 
Joined
Oct 18, 2018
Messages
969
Alright, that isn't surprising. So, lets try to import it manually to fix it and then get it done through the GUI.

What happens if you type zpool import Data-Storage? If it works you need to try the zpool remove command from above to remove that SLOG device.

If that fails, what about the output of zpool import -F -n Data-Storage?
 

1kokies

Contributor
Joined
Oct 7, 2017
Messages
138
What happens if you type zpool import Data-Storage?
output
Code:
root@freenas:~ # zpool import Data-Storage
The devices below are missing or corrupted, use '-m' to import the pool anyway:
            gptid/d3317a51-d4e1-11e7-8c89-086266295468 [log]

cannot import 'Data-Storage': one or more devices is currently unavailable
root@freenas:~ #


What happens if you type zpool import Data-Storage? If it works you need to try the zpool remove command from above to remove that SLOG device.
output, not sure what 'begin with a letter refers to'
Code:
root@freenas:~ # zpool remove 1100341364746838742
cannot open '1100341364746838742': name must begin with a letter
root@freenas:~ #


If that fails, what about the output of zpool import -F -n Data-Storage?

output=nothing happened, returns back to prompt
Code:
root@freenas:~ # zpool import -F -n Data-Storage
root@freenas:~ #


output of all 3 commands
Code:
root@freenas:~ # zpool import Data-Storage
The devices below are missing or corrupted, use '-m' to import the pool anyway:
            gptid/d3317a51-d4e1-11e7-8c89-086266295468 [log]

cannot import 'Data-Storage': one or more devices is currently unavailable
root@freenas:~ #
root@freenas:~ # zpool remove 1100341364746838742
cannot open '1100341364746838742': name must begin with a letter
root@freenas:~ #
root@freenas:~ # zpool import -F -n Data-Storage
root@freenas:~ #
 
Joined
Oct 18, 2018
Messages
969
Sorry, the remove command had a typo, you need to do zpool remove Data-Storage 1100341364746838742. I suspect that won't work either. How about zpool import -m Data-Storage? If that works, give me the output of zpool status Data-Storage
 

1kokies

Contributor
Joined
Oct 7, 2017
Messages
138
sorry about typo :), the output
Code:
root@freenas:~ # zpool remove Data-Storage 1100341364746838742
cannot open 'Data-Storage': no such pool
root@freenas:~ # zpool import -F -n Data-Storage
root@freenas:~ #
 
Top