Are 2 of My Drives Failed? (See Edit: Moving Data Onto To New Vdev, To Remove Old)

[joeschmuck]

I don't see any obvious failures so I say let it ride. I do not like the fact of that error message and I'd report it if I had this issue. But if the drives resilvered, they are in the pool and operating without any complaints, I have no idea if there is anything to do.

 

[Davvo]

He wants to remove a VDEV of the smaller drives now that he added the big ones.

 

[joeschmuck]

He wants to remove a VDEV of the smaller drives now that he added the big ones.

Thanks, guess I missed that. This thread is quite long now.

 

[isopropyl]

I don't see any obvious failures so I say let it ride. I do not like the fact of that error message and I'd report it if I had this issue. But if the drives resilvered, they are in the pool and operating without any complaints, I have no idea if there is anything to do.

Yes, this is now a different topic.

Are 2 of My Drives Failed? (See Edit: Moving Data Onto To New Vdev, To Remove Old)

What are you doing when you try to detach it? Are you detaching the second one in the active Spare part of Mirror 6?

www.truenas.com

Explains my goals.

Should I maybe move this to a new thread? Didn't want to spam the forms seeing we were already discussing it here.

 

[Davvo]

Should I maybe move this to a new thread? Didn't want to spam the forms seeing we were already discussing it here.

Not the worst idea. Before that, maybe @sretalla can help?

 

[Jailer]

Good lord! 3k views, 7 pages and 124 replies and this still isn't resolved yet?????

 

[Davvo]

Good lord! 3k views, 7 pages and 124 replies and this still isn't resolved yet?????

Well, the original issue is indeed resolved. Then... things came out along the way (there is even another thread).

 

[isopropyl]

Edited the title and original post to hopefully help reflect things.
If worse comes to worse, I will make a new thread.

But for anyone new, basically post #106 explains what I am trying to achieve, my concerns, and the posts below that show I encounter an error with it, whether it is being done correctly or not, I don't know. I suggest reading post #106 and then the posts after it for a solid grasp as it may be a bit confusing.
Thank you again, I know it's kinda a pain, I appreciate the help from the community

 

[Davvo]

Post #106

Are 2 of My Drives Failed? (See Edit: Moving Data Onto To New Vdev, To Remove Old)

What are you doing when you try to detach it? Are you detaching the second one in the active Spare part of Mirror 6?

www.truenas.com

 

[isopropyl]

Still looking for some help here

 

[Davvo]

Did you try rebooting and trying a second time?

 

[joeschmuck]

I'm going to be honest here, while I don't have much experience adding and removing VDEVs, I'm also not sure of the configuration of the entire pool.

I'm probably being stupid here but why didn't you just resilver in the 20TB drives to replace the 4TB drives? You can replace one drive in mirror1, resilver it, once done replace the second drive in mirror-1, let it resilver. Then just add a third 20tb drive (extend) to mirror-1 and now you have a 3-way mirror for mirror-1. Make sure the drives removed have been wiped for reuse.

I would also remove the spare drives and use those to bolster the other 4TB drives, add on to mirror and one to mirror-3. Next you have two more drives left over from mirror-1 and add one to mirror4 and ad the last to mirror-5. This leaves you hanging with mirror6 having a 2 way mirror of 4TB drives.

As you purchase more 20TB drives, replace the mirrored drives one at a time, not as one group.

And you can remove a mirror vdev of course via the GUI. I just created a VM of Scale and added and removed virtual drives in mirrors to test it out. Since you are not understanding what is being told to you, I highly recommend you run up a virtual machine of TrueNAS Scale so you can play with it and understand what you need to do. It's not complicated and if you are concerned about data loss, backup your data first. We always tell people this because while things generally work fine, it's also easy to click the wrong button and make a huge mistake.

This reference in the user guide should help you a lot and the VM of TrueNAS.

 

[isopropyl]

Did you try rebooting and trying a second time?

Yes. I am still concerned about the process it is attempting either way, because it is suggesting it is trying to wipe the drives.

I'm going to be honest here, while I don't have much experience adding and removing VDEVs, I'm also not sure of the configuration of the entire pool.

I'm probably being stupid here but why didn't you just resilver in the 20TB drives to replace the 4TB drives? You can replace one drive in mirror1, resilver it, once done replace the second drive in mirror-1, let it resilver. Then just add a third 20tb drive (extend) to mirror-1 and now you have a 3-way mirror for mirror-1. Make sure the drives removed have been wiped for reuse.

I would also remove the spare drives and use those to bolster the other 4TB drives, add on to mirror and one to mirror-3. Next you have two more drives left over from mirror-1 and add one to mirror4 and ad the last to mirror-5. This leaves you hanging with mirror6 having a 2 way mirror of 4TB drives.

As you purchase more 20TB drives, replace the mirrored drives one at a time, not as one group.

And you can remove a mirror vdev of course via the GUI. I just created a VM of Scale and added and removed virtual drives in mirrors to test it out. Since you are not understanding what is being told to you, I highly recommend you run up a virtual machine of TrueNAS Scale so you can play with it and understand what you need to do. It's not complicated and if you are concerned about data loss, backup your data first. We always tell people this because while things generally work fine, it's also easy to click the wrong button and make a huge mistake.

This reference in the user guide should help you a lot and the VM of TrueNAS.

So you're saying, replace 1 of the 4tb drives with a 20tb, let it resilver, then replace the other 4tb drive, then I will have replaced one of the mirrors with two 20tb drives. Then add add the 3rd onto the mirror.
And then do the same with another one of the mirrors?

And yes I did plan to remove the spare drives and add them onto the leftover 2 way 4tb mirrors to turn them into 3 way mirrors.

I guess that makes sense. I've done drive replacements so I guess I am more confident in that method.

The only reason I did it the way I did was because that was how the process was suggested. However, it does not seem to be working as intended, and even if I do it in a VM I don't know if I trust it on my live data here. I don't see any other way of doing it than what I already did for that method.

Also this is on TrueNAS Core, so maybe it is a bug on Core, and it works on Scale. Idk.vBut I will just hold and replace the drives individually. It feels a bit safer.
Unfortunately I can't really use my current 20tb drives. But it's fine, they will give me more storage.

---

One of them is already saying pre-failing, but also old_age? lmao so maybe I'll have to RMA that? The results are really weird though..?
Rerunning a scrub to see if anything changes.

Code:

Device: /dev/da17 [SAT], failed to read SMART Attribute Data.​

Code:

# smartctl -a /dev/da17
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     ST20000NM007D-3DJ103
Serial Number:    ZVT5JR8S
LU WWN Device Id: 5 000c50 0e5cb06f8
Firmware Version: SN01
User Capacity:    20,000,588,955,648 bytes [20.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Sep 20 14:49:38 2023 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  567) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        (1714) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   084   064   044    Pre-fail  Always       -       244122256
  3 Spin_Up_Time            0x0003   094   094   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       5
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   071   060   045    Pre-fail  Always       -       11347170
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       591
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       5
 18 Unknown_Attribute       0x000b   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   063   048   000    Old_age   Always       -       37 (Min/Max 28/43)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       4
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       1059
194 Temperature_Celsius     0x0022   037   047   000    Old_age   Always       -       37 (0 24 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0023   100   100   001    Pre-fail  Always       -       0
240 Head_Flying_Hours       0x0000   100   100   000    Old_age   Offline      -       311 (62 151 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       2094009876
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       549859027646

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       577         -
# 2  Short offline       Completed without error       00%       553         -
# 3  Short offline       Completed without error       00%       529         -
# 4  Short offline       Completed without error       00%       505         -
# 5  Short offline       Completed without error       00%       481         -
# 6  Short offline       Completed without error       00%       457         -
# 7  Short offline       Completed without error       00%       433         -
# 8  Short offline       Completed without error       00%       409         -
# 9  Short offline       Completed without error       00%       385         -
#10  Short offline       Completed without error       00%       361         -
#11  Short offline       Completed without error       00%       338         -
#12  Short offline       Completed without error       00%       316         -
#13  Short offline       Completed without error       00%       292         -
#14  Short offline       Completed without error       00%       268         -
#15  Short offline       Completed without error       00%       244         -
#16  Short offline       Completed without error       00%       220         -
#17  Short offline       Completed without error       00%       196         -
#18  Short offline       Completed without error       00%       172         -
#19  Short offline       Completed without error       00%       148         -
#20  Short offline       Completed without error       00%       124         -
#21  Short offline       Completed without error       00%       100         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing

 

[Davvo]

I'm probably being stupid here but why didn't you just resilver in the 20TB drives to replace the 4TB drives?

Iirc it was to put the old drives under as little stress as possible, but I might remember wrong.

 

[joeschmuck]

So you're saying, replace 1 of the 4tb drives with a 20tb, let it resilver, then replace the other 4tb drive, then I will have replaced one of the mirrors with two 20tb drives. Then add add the 3rd onto the mirror.
And then do the same with another one of the mirrors?

Yes, but there is more than one way to reach your goal is my point. This is just the method I would use as I'm upgrading my drives if I were in your situation. Another way would be to see if you can add a 20TB drive into the two drive mirror to see if it will add it, then you have a 3 way mirror, next you would replace the 4TB drives one at a time and let each resilver. Once done you should have three 20TB mirror.

Of course the safest thing to do is backup your important data first, I suspect you have said you have done that already, just 7 pages is a bit much to scan through again.

Also, try to create a VM of TrueNAS if you have a machine you can do that on. Simulate your drive configuration, maybe not all the mirrors but two sets, then add a drive that is larger, see what happens. You can then replace the smaller drives with simulated larger drives. It all should work fine but this will give you some confidence in what you are doing. It really is helpful when you are just unsure.

 

[isopropyl]

Iirc it was to put the old drives under as little stress as possible, but I might remember wrong.

Wouldn't it still be resilvering the data onto the new drives no matter what?

Yes, but there is more than one way to reach your goal is my point. This is just the method I would use as I'm upgrading my drives if I were in your situation. Another way would be to see if you can add a 20TB drive into the two drive mirror to see if it will add it, then you have a 3 way mirror, next you would replace the 4TB drives one at a time and let each resilver. Once done you should have three 20TB mirror.

Of course the safest thing to do is backup your important data first, I suspect you have said you have done that already, just 7 pages is a bit much to scan through again.

Also, try to create a VM of TrueNAS if you have a machine you can do that on. Simulate your drive configuration, maybe not all the mirrors but two sets, then add a drive that is larger, see what happens. You can then replace the smaller drives with simulated larger drives. It all should work fine but this will give you some confidence in what you are doing. It really is helpful when you are just unsure.

Yes, I understand there is more than one way. I meant that the method you suggested was not apparent to me.
That is actually smart though, to attach the 20tb drive first onto the 2-way mirror. Lowers the risk.

And yes, my important data is backed up. But a large portion of my data is like a decade worth of media which would really really suck to lose. But I can get it all back over time. Really the music would suck the most tbh lmao.
And yeah, I am reasonably confident in the replacement method. It was the method of simply removing a vdev, and expecting it to migrate the data off of them, onto the new drive that I felt really uncomfortable with.

Again, it sucks because I have 3 20tb drives in the machine now that aren't exactly being utilized and are part of the pool. That being said, I will need the extra space anyways, so I'll just have to take it for what it is for now. I got the 2 spares in still for now, and I have 2 cold spares of the 4tb drives so hopefully that can hold me over. Worst case I order some more 4tb drives for cheap lmao.
I definitely should just order a few more I guess and I can just make all of them into 3-ways for now.