raidz - degraded - investiagion

[phier]

Hello,
today i received email regarding that drive was removed and raidz degraded.

The following alert has been cleared:
* Pool universalsoldier state is DEGRADED: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state.
The following devices are not healthy:

• Disk ST18000NM000J-2TV103 ZR52K9D5 is REMOVED

The dmesg log:

debian Pastezone

0)
raidZ1 was composed of 3x18tb drive; 2x wd and 1x seagate ... apparently seagate died.

1)
Is it a good idea to run smartctl long on that ata2 drive?
2)
Do i have to power off ; and replace drive immediately or can i run the machine in degraded mode for next 14 days?

Is it possible to do more further investigation? As I left home for another 14 days i cant execute any manual intervention ie cable replacement etc.

Thank you so much.

 

[Patrick M. Hausen]

Do i have to power off ; and replace drive immediately or can i run the machine in degraded mode for next 14 days?

If another drive fails your data is gone for good. Your judgement if you want to bear that risk for 14 days or not.

 

[phier]

@Patrick M. Hausen
No clue what's better to try to backup some data or to power off and plug the new one and recover the raid.

Most of data there are w/out backup.

Also is it good idea to run smarctl - t long on that drive?

 

[Patrick M. Hausen]

Also is it good idea to run smarctl - t long on that drive?

Cannot get more broken, I guess.

Perform a backup ASAP.

 

[phier]

@Patrick M. Hausen it now shows all drive connected and says UNHEALTY

but i have no clue how it got "attached?" that ada2

 

[phier]

more details:

 

[Davvo]

Please report the model of your WD drives (SMR drives are a big NO with ZFS), then follow those steps:

1. Backup your data;
2. Check power and data connections for evident damage or not properly latched cables;
3. Check the average temperature of your drives in the reporting section of the WebUI;
4. Run zpool clear universalsoldier and then run a scrub;
5. After the scrub finishes, check if any errors were found;
6. Run a long smart test on each drive.

 

[phier]

@Davvo
These are non srm,
Wd Ultrastar hc550 2x
1* seagate need to find mode name.

Isn't scrub, smartctl heavy operation... Just thinking what's the better

To backup
Or to power off till I replace that seagate drive...

Thanks

 

[Davvo]

You should follow the steps I gave you in order to evaluate your system's situation.

 

[phier]

@Davvo
Just wondering if it's safe To run such a system where 3-4times during the day drive got removed (raid gets degraded) and then back online... Each degrade/back online doest do data recovery?

Thx

 

[Davvo]

I am having a little difficulties understanding your question, but I will try answering.
It's not safe running a system that is unstable for unknown reasons and you should backup your data, as stressed out multiple times by me and @Patrick M. Hausen, as soon as possible; that being siad, having a drive being kicked out of the pool and then reintegrated 3/4 times per day is not a danger to the integrity of the data per se (the danger lies in not identifying the cause and the degraded state of the pool).

In the list I gave to you, the scrub is the most stressful operation and as such the most dangerous one.

Edited for spelling corrections.

 

[rvassar]

Davvo is giving you good advice. You haven't provided any system information, so I'm going to guess it's a cabling problem. The "ada" kind of tells me these are SATA drives. Most SATA cables are only spec'ed for 50 insert remove events over the cable's entire lifespan. Most home builders run aground here and get quite confused trying to use cables they've had in their parts stash for a decade or more. They may work fine, and fail intermittently due to angle/bend stress, changes in temperature, etc...

A couple thoughts:
1. The uncorrectable error spec for the HC550 is 1 in 10^15, which gives you some head space for UCE, but I suspect most of us here would advise against RAIDZ1. You're looking at one UCE for roughly every 120Tb, but for 32Tb of data, that's pretty thin.
2. Your pool is 97% used. This kills ZFS performance, and significantly impacts it's ability to map around errors. A ZFS pool should never exceed 80% capacity. This is kind of a penalty imposed by all the other bits of magic ZFS brings to the table.
3. Any kind of RAID is not a backup. You need to have backups!

 

[phier]

@rvassar @Davvo thanks
Sorry I was in the stress, so missed these details.

As you mentioned, all drives are SATA, i tried to cable everything with brand new cables... are there cables that guys recommend here/ie. brand? And i did insert/remove max 5 times - as it was brand new build.

The current state:
- i backed up via borg almost 5TB of data to the borgbase.com
- backup still in progress...

The most strange thing is ... that i am not getting any error as previously ie. drive detached / raidz1 degraded... it seems it "works"? As machine is 1500 km away ... noone has phy access there i am confused - what happened that i am not getting any error.


@rvassar
1)

You're looking at one UCE for roughly every 120Tb, but for 32Tb of data, that's pretty thin.

by UCE you mean? Ie we are talking about ~4 UCE for 32Tb?

The reason why i go for RAIDZ1 was that I wanted to do onsite replica via ZFS snaps to phy different machine; and also one more borg backup to offsite cloud. That was the reason why i somehow came up to the conclusion that RAIDZ2 is not needed (ie parity on 2 drives, raid6).
Maybe that thinking was not correct at all?

2) yeah, the idea was to "clean up some data" and leave it to 80-85%

3) agree; my fault - work in progress

Thank you!

 

[Davvo]

The most strange thing is ... that i am not getting any error as previously ie. drive detached / raidz1 degraded... it seems it "works"? As machine is 1500 km away ... noone has phy access there i am confused - what happened that i am not getting any error.

Could have been a base sector in the drive, that's been mapped and replaced with a spare one by the drive's firmware, but until we get the smart data it's just a conjecture.

1)by UCE you mean? Ie we are talking about ~4 UCE for 32Tb?

The reason why i go for RAIDZ1 was that I wanted to do onsite replica via ZFS snaps to phy different machine; and also one more borg backup to offsite cloud. That was the reason why i somehow came up to the conclusion that RAIDZ2 is not needed (ie parity on 2 drives, raid6).
Maybe that thinking was not correct at all?

Also known as URE. Please read the following resource.

Assessing the Potential for Data Loss

This guide was written to be read from top to bottom without jumps, with the intent of spreading awareness to both new and experienced users; the author of this document assumes the understanding of the concepts explained in the following...

www.truenas.com

2) yeah, the idea was to "clean up some data" and leave it to 80-85%

It will take a good while.

 

[phier]

Hello,
so i did investigation and here is the dump of smarctl -t long of all 3 drives (3 one was removed repeatedly from the pool - ada2)
also data that was resilvered 133G in 00:57:28 with 0 errors on Sun Nov 19 07:06:19 2023

Also smarctl executed on ada2 havent finished:
Self-test execution status: ( 33) The self-test routine was interrupted
by the host with a hard or soft reset.

smartctl results:
ada0: http://dpaste.com//9X4AENNCQ
ada1: http://dpaste.com//E4JQEE8KL
ada2: http://dpaste.com//ARHV3MZ3V


Currently drive is not removed from the pool anymore...
So i am wondering if i have to replace it or keep it as it is.

Thank you!


pool: universalsoldier
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: resilvered 133G in 00:57:28 with 0 errors on Sun Nov 19 07:06:19 2023
config:

NAME STATE READ WRITE CKSUM
universalsoldier ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/a691e97e-ffaa-11ec-aba1-000c29f60559 ONLINE 0 0 174
gptid/a68cf3cc-ffaa-11ec-aba1-000c29f60559 ONLINE 0 0 144
gptid/a696a973-ffaa-11ec-aba1-000c29f60559 ONLINE 0 0 0

errors: No known data errors
root@truenas[~]#

 

[Davvo]

Show : for posterity

Code:

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p1 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Ultrastar DC HC550
Device Model:     WDC  WUH721818ALE6L4
Serial Number:    4ZG5509V
LU WWN Device Id: 5 000cca 2a6c258a9
Firmware Version: PCGNW232
User Capacity:    18,000,207,937,536 bytes [18.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 published, ANSI INCITS 529-2018
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Nov 25 00:27:53 2023 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (  101) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      (1973) minutes.
SCT capabilities:            (0x003d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   001    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   136   136   054    Pre-fail  Offline      -       96
  3 Spin_Up_Time            0x0007   083   083   001    Pre-fail  Always       -       342 (Average 350)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       34
  5 Reallocated_Sector_Ct   0x0033   100   100   001    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   001    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   140   140   020    Pre-fail  Offline      -       15
  9 Power_On_Hours          0x0012   099   099   000    Old_age   Always       -       11012
 10 Spin_Retry_Count        0x0013   100   100   001    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       31
 22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       794
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       794
194 Temperature_Celsius     0x0002   063   063   000    Old_age   Always       -       32 (Min/Max 21/47)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   100   100   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     10998         -
# 2  Extended offline    Completed without error       00%       416         -
# 3  Short offline       Completed without error       00%        74         -
# 4  Extended offline    Completed without error       00%        27         -
# 5  Extended offline    Interrupted (host reset)      90%         2         -
# 6  Extended offline    Aborted by host               90%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p1 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Ultrastar DC HC550
Device Model:     WDC  WUH721818ALE6L4
Serial Number:    2JGE54MC
LU WWN Device Id: 5 000cca 2abc5fc27
Firmware Version: PCGNW680
User Capacity:    18,000,207,937,536 bytes [18.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 published, ANSI INCITS 529-2018
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Nov 25 00:27:58 2023 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (  101) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      (1876) minutes.
SCT capabilities:            (0x003d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   001    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   149   149   054    Pre-fail  Offline      -       44
  3 Spin_Up_Time            0x0007   083   083   001    Pre-fail  Always       -       340 (Average 348)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       32
  5 Reallocated_Sector_Ct   0x0033   100   100   001    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   001    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   140   140   020    Pre-fail  Offline      -       15
  9 Power_On_Hours          0x0012   099   099   000    Old_age   Always       -       10807
 10 Spin_Retry_Count        0x0013   100   100   001    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       29
 22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       782
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       782
194 Temperature_Celsius     0x0002   063   063   000    Old_age   Always       -       32 (Min/Max 21/47)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   100   100   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     10795         -
# 2  Extended offline    Completed without error       00%       239         -
# 3  Extended offline    Completed without error       00%        26         -
# 4  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p1 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     ST18000NM000J-2TV103
Serial Number:    ZR52K9D5
LU WWN Device Id: 5 000c50 0dbaf74a9
Firmware Version: SC02
User Capacity:    18,000,207,937,536 bytes [18.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Nov 25 00:28:01 2023 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (  33)    The self-test routine was interrupted
                    by the host with a hard or soft reset.
Total time to complete Offline
data collection:         (  559) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      (1523) minutes.
Conveyance self-test routine
recommended polling time:      (   2) minutes.
SCT capabilities:            (0x70bd)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   083   064   044    Pre-fail  Always       -       178218896
  3 Spin_Up_Time            0x0003   094   088   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       357
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   087   060   045    Pre-fail  Always       -       503814110
  9 Power_On_Hours          0x0032   088   088   000    Old_age   Always       -       11013
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       354
 18 Unknown_Attribute       0x000b   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       8590065666
190 Airflow_Temperature_Cel 0x0022   072   048   000    Old_age   Always       -       28 (Min/Max 27/32)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       337
193 Load_Cycle_Count        0x0032   096   096   000    Old_age   Always       -       8002
194 Temperature_Celsius     0x0022   028   048   000    Old_age   Always       -       28 (0 21 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0023   100   100   001    Pre-fail  Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       10305 (58 253 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       44383245154
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       2832061906781

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Interrupted (host reset)      00%     10963         -
# 2  Short offline       Completed without error       00%     10935         -
# 3  Extended offline    Interrupted (host reset)      00%     10798         -
# 4  Extended offline    Completed without error       00%       446         -
# 5  Conveyance offline  Completed without error       00%        75         -
# 6  Short offline       Completed without error       00%        74         -
# 7  Short offline       Completed without error       00%        74         -
# 8  Extended offline    Completed without error       00%        24         -
# 9  Extended offline    Aborted by host               90%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

4ZG5509V looks great.
2JGE54MC looks great.
ZR52K9D5 lacks a recently completed long test. Please run one, let it finish, and post the results; from what I can see there is nothing wrong there, but it's old data. If it isn't able to finish one, trash it.

Note: please avoid hosting things outside the forum since they might not be around anymore when someone in the future reads the topic.

 

[phier]

@Davvo thanks for a check,
i executed it again, lets see how that ends up.

I already backed up NAS; but still not sure... regardin the data.. even i cant finish smartctl and drive is part of the POOL ... it means it can be broken?
Didnt that fact somehow "break" current data stored in pool?

thanks

 

[Davvo]

@Davvo thanks for a check,
i executed it again, lets see how that ends up.

I already backed up NAS; but still not sure... regardin the data.. even i cant finish smartctl and drive is part of the POOL ... it means it can be broken?
Didnt that fact somehow "break" current data stored in pool?

thanks

In a RAIDZ1 VDEV you have one drive of parity, meaning you can lose a single drive without issues.

Please read the following resource.

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

 

[phier]

@Davvo
hello,
seems smartctl long finished on that seagate drive

dpaste: 9HW9HBWSC

dpaste.com

also question why previous test was interrupted, if its drive/mainboard (sata) or last resort - cable (which was brand new?)

thanks!

 

[Davvo]

Show : For posterity

Code:

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p1 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     ST18000NM000J-2TV103
Serial Number:    ZR52K9D5
LU WWN Device Id: 5 000c50 0dbaf74a9
Firmware Version: SC02
User Capacity:    18,000,207,937,536 bytes [18.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Nov 26 12:53:09 2023 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (  559) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      (1523) minutes.
Conveyance self-test routine
recommended polling time:      (   2) minutes.
SCT capabilities:            (0x70bd)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   084   064   044    Pre-fail  Always       -       230858432
  3 Spin_Up_Time            0x0003   094   088   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       357
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   087   060   045    Pre-fail  Always       -       514091922
  9 Power_On_Hours          0x0032   088   088   000    Old_age   Always       -       11049
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       354
 18 Unknown_Attribute       0x000b   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       8590065666
190 Airflow_Temperature_Cel 0x0022   072   048   000    Old_age   Always       -       28 (Min/Max 27/32)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       337
193 Load_Cycle_Count        0x0032   096   096   000    Old_age   Always       -       8011
194 Temperature_Celsius     0x0022   028   048   000    Old_age   Always       -       28 (0 21 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0023   100   100   001    Pre-fail  Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       10341 (24 201 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       44410957658
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       2832086833813

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     11038         -
# 2  Extended offline    Interrupted (host reset)      00%     10963         -
# 3  Short offline       Completed without error       00%     10935         -
# 4  Extended offline    Interrupted (host reset)      00%     10798         -
# 5  Extended offline    Completed without error       00%       446         -
# 6  Conveyance offline  Completed without error       00%        75         -
# 7  Short offline       Completed without error       00%        74         -
# 8  Short offline       Completed without error       00%        74         -
# 9  Extended offline    Completed without error       00%        24         -
#10  Extended offline    Aborted by host               90%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

ZR52K9D5 looks ok.
This means the issue is likely not on the drives; please provide your full hardware specs and, as I previously wrote, please:

1. Check power and data connections for evident damage or not properly latched cables;
2. Run zpool clear universalsoldier and then run a scrub;
3. After the scrub finishes, check if any errors were found;

If you are not able to do step 1, temporarily skip it.