Failed HDD in vdev caused boot-pool error- something to worry about?

[Dopamin3]

I noticed yesterday I wasn't able to SSH or access WebGUI on my TrueNAS Scale 13.0-U4 server. But the weird thing was all my jails and virtual machines were still functioning. I walked over to the server and saw "Solaris: WARNING: Pool 'freenas-boot' has encountered an uncorrectable I/O failure and has been suspended." The boot pool is a single Samsung PM961 MLC 128GB NVME SSD.

After rebooting, I discovered one of my hard drives in my vdev died so I replaced it and am resilvering now. But server seems fully operational and fine. Is it possible the boot drive is fine and this just happened because of a "hiccup" when the one hard drive died? Or should I go ahead and replace it now to be safe? Below is output of smartctl -A on the drive:

Code:

root@RyzenNAS:~ # smartctl -A /dev/nvme0
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        34 Celsius
Available Spare:                    100%
Available Spare Threshold:          5%
Percentage Used:                    0%
Data Units Read:                    2,758,806 [1.41 TB]
Data Units Written:                 161,491 [82.6 GB]
Host Read Commands:                 42,323,790
Host Write Commands:                6,575,997
Controller Busy Time:               57
Power Cycles:                       200
Power On Hours:                     27,565
Unsafe Shutdowns:                   174
Media and Data Integrity Errors:    0
Error Information Log Entries:      47
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               34 Celsius
Temperature Sensor 2:               51 Celsius

 

[sretalla]

I think you want smartctl -x or -a, not -A ... those should give you the details of the 47 errors in the log.

 

[WI_Hedgehog]

=== START OF SMART DATA SECTION ===
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 34 Celsius
Temperature Sensor 1: 34 Celsius
Temperature Sensor 2: 51 Celsius
[/CODE]

51C? What's that about?
No vendor or model information reported?

 

[winnielinnie]

Since this is an NVMe drive, you might get better information with nvmecontrol.

Examples to check the logs of errors, SMART, and selftests (respectively pages 1, 2, and 6):

Code:

nvmecontrol logpage -p 1 nvme0
nvmecontrol logpage -p 2 nvme0
nvmecontrol logpage -p 6 nvme0

To run a short and long selftest (respectively):

Code:

nvmecontrol selftest -c 1 nvme0
nvmecontrol selftest -c 2 nvme0

 

[Dopamin3]

Since this is an NVMe drive, you might get better information with nvmecontrol.

Examples to check the logs of errors, SMART, and selftests (respectively pages 1, 2, and 6):

Code:

nvmecontrol logpage -p 1 nvme0
nvmecontrol logpage -p 2 nvme0
nvmecontrol logpage -p 6 nvme0

To run a short and long selftest (respectively):

Code:

nvmecontrol selftest -c 1 nvme0
nvmecontrol selftest -c 2 nvme0

Thanks (also thanks to @WI_Hedgehog and @sretalla).

I ran the long selftest as well as those three initial commands and here is the output:

Code:

root@RyzenNAS:~ # nvmecontrol logpage -p 1 nvme0
Error Information Log
=====================
Entry 01
=========
 Error count:          47
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 02
=========
 Error count:          46
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 03
=========
 Error count:          45
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 04
=========
 Error count:          44
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 05
=========
 Error count:          43
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 06
=========
 Error count:          42
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 07
=========
 Error count:          41
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 08
=========
 Error count:          40
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 09
=========
 Error count:          39
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 10
=========
 Error count:          38
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 11
=========
 Error count:          37
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 12
=========
 Error count:          36
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 13
=========
 Error count:          35
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 14
=========
 Error count:          34
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 15
=========
 Error count:          33
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 16
=========
 Error count:          32
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 17
=========
 Error count:          31
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 18
=========
 Error count:          30
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 19
=========
 Error count:          29
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 20
=========
 Error count:          28
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 21
=========
 Error count:          27
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 22
=========
 Error count:          26
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 23
=========
 Error count:          25
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 24
=========
 Error count:          24
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 25
=========
 Error count:          23
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 26
=========
 Error count:          22
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 27
=========
 Error count:          21
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 28
=========
 Error count:          20
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 29
=========
 Error count:          19
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 30
=========
 Error count:          18
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 31
=========
 Error count:          17
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 32
=========
 Error count:          16
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 33
=========
 Error count:          15
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 34
=========
 Error count:          14
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 35
=========
 Error count:          13
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 36
=========
 Error count:          12
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 37
=========
 Error count:          11
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 38
=========
 Error count:          10
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 39
=========
 Error count:          9
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 40
=========
 Error count:          8
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 41
=========
 Error count:          7
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 42
=========
 Error count:          6
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 43
=========
 Error count:          5
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 44
=========
 Error count:          4
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 45
=========
 Error count:          3
 Submission queue ID:  0
 Command ID:           9
 Status:
  Phase tag:           0
  Status code:         11
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       4
 LBA:                  0
 Namespace ID:         1
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 46
=========
 Error count:          2
 Submission queue ID:  0
 Command ID:           22
 Status:
  Phase tag:           0
  Status code:         2
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       65535
 LBA:                  0
 Namespace ID:         0
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
Entry 47
=========
 Error count:          1
 Submission queue ID:  0
 Command ID:           22
 Status:
  Phase tag:           0
  Status code:         2
  Status code type:    0
  More:                1
  DNR:                 0
 Error location:       65535
 LBA:                  0
 Namespace ID:         0
 Vendor specific info: 0
 Transport type:       0
 Command specific info:0
 Transport specific:   0
root@RyzenNAS:~ # nvmecontrol logpage -p 2 nvme0
SMART/Health Information Log
============================
Critical Warning State:         0x00
 Available spare:               0
 Temperature:                   0
 Device reliability:            0
 Read only:                     0
 Volatile memory backup:        0
Temperature:                    305 K, 31.85 C, 89.33 F
Available spare:                100
Available spare threshold:      5
Percentage used:                0
Data units (512,000 byte) read: 2758806
Data units written:             161496
Host read commands:             42323805
Host write commands:            6576568
Controller busy time (minutes): 57
Power cycles:                   200
Power on hours:                 27568
Unsafe shutdowns:               174
Media errors:                   0
No. error info log entries:     47
Warning Temp Composite Time:    0
Error Temp Composite Time:      0
Temperature Sensor 1:           305 K, 31.85 C, 89.33 F
Temperature Sensor 2:           322 K, 48.85 C, 119.93 F
Temperature 1 Transition Count: 0
Temperature 2 Transition Count: 0
Total Time For Temperature 1:   0
Total Time For Temperature 2:   0
root@RyzenNAS:~ # nvmecontrol logpage -p 6 nvme0
Device Self-test Status
=======================
Current Operation: No device self-test operation in progress
Results
[ 0] Extended device self-test completed without error

 

[Dopamin3]

Bump.

Is it bad this NVME drive has 47 error counts with "Error location: 4"? System seems to work fine but I would like to be proactive and replace it if suspicious.