Cannot import existing pool after replacing faulty RAM - TrueNAS Scale 22.12.1

publicgsk

Cadet
Joined
Mar 15, 2023
Messages
4
Hi all,

I am in the process of upgrading my TrueNAS server. I had to replace the RAM as my server was freezing at random. Memtest86 helped diagnose the issue with RAM. After I turned the server off, replaced the RAM and restarted, the drives on one of my pools started showing as exported and cannot be imported. Pool is OFFLINE and drives cannot be added back to the pool.
1678947036655.png


Code:
root@truenas[~]# zpool import
   pool: pool1
     id: 1955755290058112088
  state: ONLINE
status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
 config:

        pool1                                     ONLINE
          mirror-0                                ONLINE
            4dc4c370-a486-11ea-b4e1-f48e387928a5  ONLINE
            9f743e61-87d2-11ed-a535-f48e387928a5  ONLINE
root@truenas[~]# zpool import pool1
cannot import 'pool1': pool was previously in use from another system.
Last accessed by truenas (hostid=4edf30e7) at Thu Mar  9 23:32:18 2023
The pool can be imported, use 'zpool import -f' to import the pool.
root@truenas[~]# zpool import pool1 -f
cannot import 'pool1': I/O error
        Destroy and re-create the pool from
        a backup source.
root@truenas[~]# 

One of the drives seems to have failed during the upgrade process.

Code:
root@truenas[~]# smartctl -a /dev/sdm -T verypermissive -s on
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Read Device Identity failed: scsi error unsupported field in scsi command

=== START OF INFORMATION SECTION ===
Device Model:     [No Information Found]
Serial Number:    [No Information Found]
Firmware Version: [No Information Found]
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   [No Information Found]
Local Time is:    Fri Mar 10 05:50:11 2023 EST
SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 82-83 don't show if SMART supported.
SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 85-87 don't show if SMART is enabled.
                  Checking to be sure by trying SMART RETURN STATUS command.
SMART support is: Unknown - Try option -s with argument 'on' to enable it.
=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enable failed: scsi error unsupported field in scsi command

Read SMART Data failed: scsi error unsupported field in scsi command


=== START OF READ SMART DATA SECTION ===
SMART Status command failed: scsi error unsupported field in scsi command
SMART overall-health self-assessment test result: UNKNOWN!
SMART Status, Attributes and Thresholds cannot be read.

Read SMART Error Log failed: scsi error unsupported field in scsi command

Read SMART Self-test Log failed: scsi error unsupported field in scsi command

Selective Self-tests/Logging not supported

root@truenas[~]# smartctl -a /dev/sdl                        
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Elements / My Passport (USB, AF)
Device Model:     WDC WD40NMZW-11GX6S1
Serial Number:    WD-WX21DC6PSVJ7
LU WWN Device Id: 5 0014ee 6078b5bca
Firmware Version: 01.01A01
User Capacity:    4,000,753,472,000 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 (minor revision not indicated)
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Mar 10 05:53:55 2023 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                ( 4424) seconds.
Offline data collection
capabilities:                    (0x1b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        No Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 228) minutes.
SCT capabilities:              (0x30b5) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   199   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   253   253   021    Pre-fail  Always       -       5100
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
194 Temperature_Celsius     0x0022   111   101   000    Old_age   Always       -       41

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      3393         -
# 2  Short offline       Completed without error       00%      3392         -

Selective Self-tests/Logging not supported


I am new to TrueNAS. Any help in recovering the data from one of the good disk is greatly appreciated.
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,611
Is your failing drive connected via USB?
 

publicgsk

Cadet
Joined
Mar 15, 2023
Messages
4
Is your failing drive connected via USB?
Yes. Both drives in that pool are connected via USB. One failed and the pool crashed before I could move my data to a NAS rated internal drive.
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,611
Their are lots of issues with USB, see below. But, let us concentrate and see if we can get your pool back. You can try this to see if the pool could be imported, (but this does not actually do an import);

zpool import -fFn pool1

Another trick is to potentially remove the failed drive from the USB enclosure. Then see if it can be used as a plain SATA drive. It is possible that the USB to SATA adapter failed. Or the power supply for the USB enclosure failed. And yet the SATA disk inside might be okay. Just be careful not to damage the disk. Or to use it on a different computer which might write to it.

It does appear that some of those USB drives use PC power. It is possible that your USB enclosure can't get enough power from the host computer. You might try a powered USB hub, if you have one.


Here is what I wrote about USB attached storage for ZFS data pools. Note that not all, or not even most will apply to all users. But, it is a starting point to understanding why we don't recommended them;
 

publicgsk

Cadet
Joined
Mar 15, 2023
Messages
4
Their are lots of issues with USB, see below. But, let us concentrate and see if we can get your pool back. You can try this to see if the pool could be imported, (but this does not actually do an import);

zpool import -fFn pool1

Another trick is to potentially remove the failed drive from the USB enclosure. Then see if it can be used as a plain SATA drive. It is possible that the USB to SATA adapter failed. Or the power supply for the USB enclosure failed. And yet the SATA disk inside might be okay. Just be careful not to damage the disk. Or to use it on a different computer which might write to it.

It does appear that some of those USB drives use PC power. It is possible that your USB enclosure can't get enough power from the host computer. You might try a powered USB hub, if you have one.


Here is what I wrote about USB attached storage for ZFS data pools. Note that not all, or not even most will apply to all users. But, it is a starting point to understanding why we don't recommended them;
@Arwen This information was really very useful and educational. Like you mentioned, the command did not import the pool and did not present any output on the shell either. However, I was able to run the pool in a degraded mode with the bad drive removed. All the data is still available since it was in mirror mode.
1679020296021.png


I am copying it to NAS rated internal drives now, before I try anything else with these USB drives. Once the copying is done, I will experiment with drives removed from the external cases and see if I can convert them to internal drives.
 

publicgsk

Cadet
Joined
Mar 15, 2023
Messages
4
@Arwen Thank you so much. I really appreciate your time and help.
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,611
You are welcome, and I am glad you were able to import your pool.

We also have a TrueNAS SCALE Beginner's Intro that is a useful, but short, read;

Their are plenty of Resources, on various topics, see the "Resources" link at the top of the forum pages. Some of these were copied from sticky forum posts, which may still exist too.
 
Top