Intel data centre SSD claims 2^32 media errors, SMART tests pass, zpool looks ok

[Patrick M. Hausen]

Hi all,

being made aware of the tool by another thread on this forum I just installed scrutiny for disk/ssd monitoring. We have two TrueNAS CORE systems with 6 Intel P4510 (SSDPE2KX010T8) SSDs each. Of these 12 SSDs 3 show 4294967296 (exactly 2^32) media errors in scrutiny and in smartctl, yet SMART tests pass as well as the regular pool scrubs.

Has anyone else ever seen this? I suspect a firmware problem/bug and I am tempted to simply apply the update to VDV10184 from (currently) VDV10170.

All comments appreciated. Here's the smart output for one of the SSDs:

Code:

root@freenas01[~]# smartctl -a /dev/nvme6
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPE2KX010T8
Serial Number:                      ******************
Firmware Version:                   VDV10170
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Total NVM Capacity:                 1,000,204,886,016 [1.00 TB]
Unallocated NVM Capacity:           0
Controller ID:                      0
NVMe Version:                       1.2
Number of Namespaces:               128
Local Time is:                      Fri Jan 26 15:28:38 2024 CET
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x000e):   Format Frmw_DL NS_Mngmt
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Log Page Attributes (0x0e):         Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size:         32 Pages
Warning  Comp. Temp. Threshold:     70 Celsius
Critical Comp. Temp. Threshold:     80 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    12.00W       -        -    0  0  0  0        0       0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        38 Celsius
Available Spare:                    99%
Available Spare Threshold:          10%
Percentage Used:                    19%
Data Units Read:                    341,352,361 [174 TB]
Data Units Written:                 1,402,767,925 [718 TB]
Host Read Commands:                 2,603,422,859
Host Write Commands:                51,519,608,582
Controller Busy Time:               4,272
Power Cycles:                       24
Power On Hours:                     42,659
Unsafe Shutdowns:                   14
Media and Data Integrity Errors:    4,294,967,296
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 64 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS
  0          1     2       -  0xc00c      -            0     -     -

root@freenas01[~]# 

Kind regards
Patrick

EDIT/P.S. I just checked the endurance rating: 1.92 PBW (petabytes!) for this particular model. So it's probably not the firmware telling me that I have written my share.

 

[dak180]

@Patrick M. Hausen can you show the output of smartctl -AxHi the drive or smartctl -AxHij --log="error" --log="selftest" if you can run it with smartctl 7.4?

 

[Patrick M. Hausen]

Code:

root@freenas01[~]# smartctl -AxHij --log=error --log=selftest /dev/nvme6
{
  "json_format_version": [
    1,
    0
  ],
  "smartctl": {
    "version": [
      7,
      2
    ],
    "svn_revision": "5236",
    "platform_info": "FreeBSD 13.1-RELEASE-p9 amd64",
    "build_info": "(local build)",
    "argv": [
      "smartctl",
      "-AxHij",
      "--log=error",
      "--log=selftest",
      "/dev/nvme6"
    ],
    "exit_status": 0
  },
  "device": {
    "name": "/dev/nvme6",
    "info_name": "/dev/nvme6",
    "type": "nvme",
    "protocol": "NVMe"
  },
  "model_name": "INTEL SSDPE2KX010T8",
  "serial_number": "******************",
  "firmware_version": "VDV10170",
  "nvme_pci_vendor": {
    "id": 32902,
    "subsystem_id": 32902
  },
  "nvme_ieee_oui_identifier": 6083300,
  "nvme_total_capacity": 1000204886016,
  "nvme_unallocated_capacity": 0,
  "nvme_controller_id": 0,
  "nvme_version": {
    "string": "1.2",
    "value": 66048
  },
  "nvme_number_of_namespaces": 128,
  "local_time": {
    "time_t": 1706339894,
    "asctime": "Sat Jan 27 08:18:14 2024 CET"
  },
  "smart_status": {
    "passed": true,
    "nvme": {
      "value": 0
    }
  },
  "nvme_smart_health_information_log": {
    "critical_warning": 0,
    "temperature": 38,
    "available_spare": 99,
    "available_spare_threshold": 10,
    "percentage_used": 19,
    "data_units_read": 341353046,
    "data_units_written": 1402829288,
    "host_reads": 2603508188,
    "host_writes": 51521474719,
    "controller_busy_time": 4272,
    "power_cycles": 24,
    "power_on_hours": 42676,
    "unsafe_shutdowns": 14,
    "media_errors": 4294967296,
    "num_err_log_entries": 0,
    "warning_temp_time": 0,
    "critical_comp_time": 0
  },
  "temperature": {
    "current": 38
  },
  "power_cycle_count": 24,
  "power_on_time": {
    "hours": 42676
  }
}
root@freenas01[~]#

 

[dak180]

Unfortunately the extra info afforded by smartctl -AxHi --log=error --log=selftest needs smartctl 7.3 for most of it and 7.4 for all of it; on 7.2 it will not give you any more info than smartctl -AxHi (it can actually give less info). Also, 7.4 has some bugfixes for correctly dealing with some NVMe dives that behave oddly for error reporting.

 

[Patrick M. Hausen]

Here you go - thanks for looking into it.

Code:

root@freenas01[~]# /mnt/zfs/scripts/smartctl -AxHij --log=error --log=selftest /dev/nvme6
{
  "json_format_version": [
    1,
    0
  ],
  "smartctl": {
    "version": [
      7,
      4
    ],
    "pre_release": false,
    "svn_revision": "5530",
    "platform_info": "FreeBSD 13.1-RELEASE-p9 amd64",
    "build_info": "(local build)",
    "argv": [
      "smartctl",
      "-AxHij",
      "--log=error",
      "--log=selftest",
      "/dev/nvme6"
    ],
    "exit_status": 0
  },
  "local_time": {
    "time_t": 1706377136,
    "asctime": "Sat Jan 27 18:38:56 2024 CET"
  },
  "device": {
    "name": "/dev/nvme6",
    "info_name": "/dev/nvme6",
    "type": "nvme",
    "protocol": "NVMe"
  },
  "model_name": "INTEL SSDPE2KX010T8",
  "serial_number": "******************",
  "firmware_version": "VDV10170",
  "nvme_pci_vendor": {
    "id": 32902,
    "subsystem_id": 32902
  },
  "nvme_ieee_oui_identifier": 6083300,
  "nvme_total_capacity": 1000204886016,
  "nvme_unallocated_capacity": 0,
  "nvme_controller_id": 0,
  "nvme_version": {
    "string": "1.2",
    "value": 66048
  },
  "nvme_number_of_namespaces": 128,
  "smart_support": {
    "available": true,
    "enabled": true
  },
  "smart_status": {
    "passed": true,
    "nvme": {
      "value": 0
    }
  },
  "nvme_smart_health_information_log": {
    "critical_warning": 0,
    "temperature": 38,
    "available_spare": 99,
    "available_spare_threshold": 10,
    "percentage_used": 19,
    "data_units_read": 341353181,
    "data_units_written": 1402865619,
    "host_reads": 2603525134,
    "host_writes": 51522610035,
    "controller_busy_time": 4272,
    "power_cycles": 24,
    "power_on_hours": 42686,
    "unsafe_shutdowns": 14,
    "media_errors": 4294967296,
    "num_err_log_entries": 0,
    "warning_temp_time": 0,
    "critical_comp_time": 0
  },
  "temperature": {
    "current": 38
  },
  "power_cycle_count": 24,
  "power_on_time": {
    "hours": 42686
  },
  "nvme_error_information_log": {
    "size": 64,
    "read": 16,
    "unread": 0,
    "table": [
      {
        "error_count": 1,
        "submission_queue_id": 2,
        "status_field": {
          "value": 24582,
          "do_not_retry": true,
          "status_code_type": 0,
          "status_code": 6,
          "string": "Internal Error"
        },
        "phase_tag": false,
        "lba": {
          "value": 0
        }
      }
    ]
  }
}
root@freenas01[~]#

 

[dak180]

@Patrick M. Hausen Two things: one, with 7.4 run a short and long test on the drive and two, repost the output without the j switch (while most output is represented in json unfortunately not all of it is). That said, the error log entry is not a great sign...

 

[Patrick M. Hausen]

OK, will do that. But first a summary of the situation.

This is the first system with two drives showing this error in scrutiny:


These two drives are part of a a single mirror. Nonetheless the zpool was perfectly healthy all the time. As a precautionary measure I removed this mirror vdev. The procedure took 3 minutes:

Code:

remove: Removal of vdev 2 copied 169G in 0h3m, completed on Sun Jan 28 18:17:16 2024
    33.2M memory used for removed device mappings

A scrub - just to be sure - afterwards also showed no problems:

Code:

  scan: scrub repaired 0B in 00:11:22 with 0 errors on Sun Jan 28 18:29:00 2024
remove: Removal of vdev 2 copied 169G in 0h3m, completed on Sun Jan 28 18:17:16 2024
    33.2M memory used for removed device mappings
config:

    NAME                                            STATE     READ WRITE CKSUM
    zfs                                             ONLINE       0     0     0
      mirror-0                                      ONLINE       0     0     0
        gptid/76d297b3-764e-11e9-9272-0025905f9a82  ONLINE       0     0     0
        gptid/7736690c-764e-11e9-9272-0025905f9a82  ONLINE       0     0     0
      mirror-1                                      ONLINE       0     0     0
        gptid/77bb2cdb-764e-11e9-9272-0025905f9a82  ONLINE       0     0     0
        gptid/78234e94-764e-11e9-9272-0025905f9a82  ONLINE       0     0     0

The other system has got only a single problematic SSD:


Again to be on the safe side I tried to remove the affected mirror device - to my surprise it won't let me:

Code:

root@freenas01[~]# zpool remove zfs mirror-2
cannot remove mirror-2: invalid config; all top-level vdevs must have the same sector size and not be raidz.

I'll start with the SMART tests on the system with the devices successfully removed and in parallel investigate the "sector size" issue on the other server.

Kind regards, TIA,
Patrick

 

[Patrick M. Hausen]

Now things are getting weird ...

Code:

root@freenas02[~]# /mnt/zfs/scripts/smartctl -t short /dev/nvme6
smartctl 7.4 2023-08-01 r5530 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

Self-tests not supported

root@freenas02[~]# /mnt/zfs/scripts/smartctl -t long /dev/nvme6
smartctl 7.4 2023-08-01 r5530 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

Self-tests not supported

 

[Patrick M. Hausen]

I have come to a reasonable (I think) decision. I will suggest to my CTO tomorrow to replace all SSDs with new ones. Since in the 300-400 €/$ price range the regular capacity is 3.84 TB now, to save some money we can build a new pool of a single three-way-mirror vdev, copy over the VMs and that is that. That's still roughly 33% more capacity than we have now.

I also added a replication task that pushes all the VM datasets of these systems to an internal time machine backup TrueNAS in our Karlsruhe office. We already had both servers replicating everything to each other - which was part of our "poor man's HA" concept, but better have a third copy. The target system is storage only, spinning disks, so ...

Still I am very curious what the hell is going on, here.

Kind regards,
Patrick

 

[Patrick M. Hausen]

And the solution to the "sector size" problem on freenas01:

Code:

root@freenas01[~]# zdb -U /data/zfs/zpool.cache
zfs:
    version: 5000
    name: 'zfs'
    [...]
    vdev_children: 3
    [...]
        children[0]:
            type: 'mirror'
            id: 0
            guid: 10499421824644026476
            metaslab_array: 50
            metaslab_shift: 33
            ashift: 12
            [...]
        children[1]:
            type: 'mirror'
            id: 1
            guid: 15871660582752688154
            metaslab_array: 42
            metaslab_shift: 33
            ashift: 12
            [...]
        children[2]:
            type: 'mirror'
            id: 2
            guid: 1097544639037701444
            metaslab_array: 374
            metaslab_shift: 33
            ashift: 9
            [...]

Now how on earth did that come to pass?

 

[dak180]

Now things are getting weird ...

Code:

root@freenas02[~]# /mnt/zfs/scripts/smartctl -t short /dev/nvme6
smartctl 7.4 2023-08-01 r5530 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

Self-tests not supported

root@freenas02[~]# /mnt/zfs/scripts/smartctl -t long /dev/nvme6
smartctl 7.4 2023-08-01 r5530 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

Self-tests not supported

Self tests are optional for drives to support for NVMe versions less than 1.4 if I recall correctly (support for this is one of the things not exposed in json output).

Still I am very curious what the hell is going on, here.

As am I.

And the solution to the "sector size" problem on freenas01:

Code:

root@freenas01[~]# zdb -U /data/zfs/zpool.cache
zfs:
    version: 5000
    name: 'zfs'
    [...]
    vdev_children: 3
    [...]
        children[0]:
            type: 'mirror'
            id: 0
            guid: 10499421824644026476
            metaslab_array: 50
            metaslab_shift: 33
            ashift: 12
            [...]
        children[1]:
            type: 'mirror'
            id: 1
            guid: 15871660582752688154
            metaslab_array: 42
            metaslab_shift: 33
            ashift: 12
            [...]
        children[2]:
            type: 'mirror'
            id: 2
            guid: 1097544639037701444
            metaslab_array: 374
            metaslab_shift: 33
            ashift: 9
            [...]

Now how on earth did that come to pass?

The mismatched ashift could be from a number of things; one example: if the reported sector sizes of the disks of the mirror are 512(e) and the mirror was created with a sufficiently old zfs/freenas version (do not recall exactly when this changed) the default setting from the gui would give it an ashift value of 9, but later versions default to 12 for everything.

I have come to a reasonable (I think) decision. I will suggest to my CTO tomorrow to replace all SSDs with new ones.

I would suggest ones that can run self tests and I would also recomend (assuming you do not already have something like this) my TrueNAS Report for ongoing monitoring.