What does critical error "Currently unreadable (pending) sectors" mean?

[Ericloewe]

I swear, these %@#$|(! moderators cause like 70% of thread derailments! They should do something about these idiots!

In all seriousness, though, I'm not a happy camper because:

• Some of my planning relies on the availability of 4 TB SATA SSDs, due to poor decision-making that predates me and cannot be changed at the moment.
• I can't just bite the bullet and migrate to NVMe wholesale - I have a bunch of 10-bay Dell R630s, which can only take 4 NVMe disks. And worse, a few IBM/Lenovo x3550 m5s that don't support U.2 disks at all.
• I'd planned on the availability of SSDs from three major vendors: Crucial/Micron, Samsung and SanDisk/WD (Kioxia, aka Toshiba never had a good showing in the SATA market and they seem to be NVMe-onlyt these days). That leaves:
  • Crucial MX500: Ridiculous long-standing firmware bug whereby TRIM causes the drive to go ballistic and start rewriting crap like there's no tomorrow. The result is monstrous write amplification.
  • Micron enterprise stuff: Limited availability, terrible pricing.
  • Samsung 870 EVO: At least a bad batch, possibly a firmware issue?
  • Samsung 870 QVO: Probably the same terribleness, but with the "benefits" of QLC.
  • WD Blue SATA: A bit too close to the WD Green, whose TRIM bug corrupted data. Although maybe they've since changed the controllers two or three times and the NAND four times. I have a bunch of them installed, too, no issues so far.
  • WD Red SATA: Same as Blue, with more Red. Also more cost.
• I'm sick of dealing with storage issues:
  • Gluster is a massive pain in the ass and I'm stuck with it due to the requirements and aforementioned bad decisions.
  • An old Gluster volume was hanging on by a thread, between Gluster bugs, server failures, and god-awful slow 2 TB 2.5" SMR disks (because enough storage had to be crammed into 2.5" drive bays and nobody realized the disks were SMR).
  • And now the SSDs are pissing me off as well!

What saved my ass from a lot of pain this week was, as usual, ZFS. With literal hundreds of thousands of checksum errors, I could've kissed this data goodbye with any other filesystem. At one point, the second disk was faulted and there was even a corrupted file, but that actually got fixed with the resilver after the zpool clear, so users managed to just do their normal work through the ordeal.

So what will you be replacing these with? The RMA'd replacements?

Well, I'm minus my spare now, so I definitely need to buy more. I'm leaning towards running one WD Blue and one Samsung 870 EVO (from the RMA), to spread out the risk a bit. And despite their association with the crap low-end buggy SSDs, the WD Blues are the most reliable so far, so I'll probably buy more of those.

 

[MikeyG]

I have "so many" of the WD blues that have been working well, but this is exacerbating my fear of the day when I have to replace them with a different model. Do we have anyplace official that tracks what SSD's have a track record of working, or recommendations for SSD's that are not necessarily enterprise grade?

 

[Brian Stretch]

Some of my planning relies on the availability of 4 TB SATA SSDs, due to poor decision-making that predates me and cannot be changed at the moment.

• I can't just bite the bullet and migrate to NVMe wholesale - I have a bunch of 10-bay Dell R630s, which can only take 4 NVMe disks. And worse, a few IBM/Lenovo x3550 m5s that don't support U.2 disks at all.

Maybe 3-4 15.36TB NVMe drives in RAIDZ1 or mirrored? There are deals to be had on Gen3 drives now that PCIe Gen4 and 5 are out. Cost per TB is favorable vs smaller drives.

My aging 4x1.92TB Micron 1100 RAIDZ1 is still chugging along, though I did replace 1 drive after TrueNAS complained. I don't need more space or speed but I've been thinking about enterprise-grade NVMe for reliability. Can't justify it yet.

 

[Ericloewe]

15.36TB NVMe

That's an option of course, but the cost rises significantly. I'd also have to make sure all the R630s are actually wired up for U.2 support (if not, it's four more SFF-8643 cables, plus a retimer board, per server).

 

[Etorix]

I swear, these %@#$|(! moderators cause like 70% of thread derailments! They should do something about these idiots!

Who moderates the moderators?

I'd planned on the availability of SSDs from three major vendors: Crucial/Micron, Samsung and SanDisk/WD (Kioxia, aka Toshiba never had a good showing in the SATA market and they seem to be NVMe-onlyt these days).

Kioxia still has a SAS SSD line. That might not help if an extra HBA is needed, tough.

Kioxia PM7 24G SAS4 SSDs Launched

The new Kioxia PM7 SSDs reach up to 30.72TB per drive and utilize the new 24G SAS4 interface for next-generation storage arrays and servers

www.servethehome.com

 

[joeschmuck]

Who moderates the moderators?

Eh, keeps things spicy.

 

[Ericloewe]

Who moderates the moderators?


Kioxia still has a SAS SSD line. That might not help if an extra HBA is needed, tough.

Kioxia PM7 24G SAS4 SSDs Launched

The new Kioxia PM7 SSDs reach up to 30.72TB per drive and utilize the new 24G SAS4 interface for next-generation storage arrays and servers

www.servethehome.com

SAS achieves the feat of having zero redeeming qualities relative to NVMe, though I do have the HBAs and expanders already in place, which makes it less horrible. Still, I suspect pricing will be somewhat worse than NVMe. Not that I can do much about it if I need SATA or SAS…

 

[gameoffuture]

I have "so many" of the WD blues that have been working well, but this is exacerbating my fear of the day when I have to replace them with a different model. Do we have anyplace official that tracks what SSD's have a track record of working, or recommendations for SSD's that are not necessarily enterprise grade?

I am currently looking for 2.5 inch drives for TrueNAS, and I am looking for options. I bought a pair of WD Blues (SA510) after reading this, and maybe you have the older parts or firmware in there? Mine are horrible, and failing.

I just want to report that WD Blues have failed miserably within the past 6 months (1 failed overnight, got replaced, then new one failed 10 weeks later the exact same way). WD Blue throws random IO errors that may go away after a reboot, but then they die completely (not detected in BIOS at all). This is newer batch/part number which seems to have firmware problems according to WD support and many Reddit posts. Additionally WD Dashboard does not detect the drive in Windows if you have ext4 or ZFS partitons, so you have to wipe the drives, then it will detect them, so firmware update after deployment is a pain also. Just to add insult to injury it will cost me $20 to ship a $65 500GB drive back to WD for RMA, for it to die again in few weeks, so I'm not even bothering with RMA anymore to cut my losses.

I have tested a pair of each WD Blue SA510 drives (500GB) and Crucial MX500 (1TB), and just purchased two Samsung 870 (1TB) that I have not tested yet. I have not installed these in my TrueNAS yet, because I wanted to see if there are any showstoppers first. I used them in ZFS mirroring pairs to run VMs off of them on Proxmox systems.

I have not seen any errors with MX500s yet, because they are only a few weeks old. So I assume they are better than WD Blue so far, that at least their controllers are noit throwing IO errors yet.

What is the current state of MX500 in 2023 (because things obviously change)? Is this MX500 issue addressed yet in newer firmware, or I'm just lucky or or should wait longer?
I know this is somewhat off-topic, but I need to buy drives for whatever that will replace my current TrueNAS system that has been running 24/7 since 2009-ish soon (it is a Core 2 duo 6300, with 6GB of RAM, not sure yet if it has had enough, or "old is gold?").
what are good options in 2023? I'm looking for 2.5 inch drives, SSD and HDD SATA.

 

[Ericloewe]

Is this MX500 issue addressed yet in newer firmware

I second this question. I was looking around the other day and found very little new information.

I know this is somewhat off-topic, but I need to buy drives for whatever that will replace my current TrueNAS system that has been running 24/7 since 2009-ish soon (it is a Core 2 duo 6300, with 6GB of RAM, not sure yet if it has had enough, or "old is gold?").
what are good options in 2023? I'm looking for 2.5 inch drives, SSD and HDD SATA.

See my sig for starting points. You're in desperate need of an upgrade.

 

[MikeyG]

I am currently looking for 2.5 inch drives for TrueNAS, and I am looking for options. I bought a pair of WD Blues (SA510) after reading this, and maybe you have the older parts or firmware in there? Mine are horrible, and failing.

I just want to report that WD Blues have failed miserably within the past 6 months (1 failed overnight, got replaced, then new one failed 10 weeks later the exact same way). WD Blue throws random IO errors that may go away after a reboot, but then they die completely (not detected in BIOS at all). This is newer batch/part number which seems to have firmware problems according to WD support and many Reddit posts. Additionally WD Dashboard does not detect the drive in Windows if you have ext4 or ZFS partitons, so you have to wipe the drives, then it will detect them, so firmware update after deployment is a pain also. Just to add insult to injury it will cost me $20 to ship a $65 500GB drive back to WD for RMA, for it to die again in few weeks, so I'm not even bothering with RMA anymore to cut my losses.

I have tested a pair of each WD Blue SA510 drives (500GB) and Crucial MX500 (1TB), and just purchased two Samsung 870 (1TB) that I have not tested yet. I have not installed these in my TrueNAS yet, because I wanted to see if there are any showstoppers first. I used them in ZFS mirroring pairs to run VMs off of them on Proxmox systems.

I have not seen any errors with MX500s yet, because they are only a few weeks old. So I assume they are better than WD Blue so far, that at least their controllers are noit throwing IO errors yet.

What is the current state of MX500 in 2023 (because things obviously change)? Is this MX500 issue addressed yet in newer firmware, or I'm just lucky or or should wait longer?
I know this is somewhat off-topic, but I need to buy drives for whatever that will replace my current TrueNAS system that has been running 24/7 since 2009-ish soon (it is a Core 2 duo 6300, with 6GB of RAM, not sure yet if it has had enough, or "old is gold?").
what are good options in 2023? I'm looking for 2.5 inch drives, SSD and HDD SATA.

That is not comforting to me. I last bought a WD Blue drive about a year ago. I've been running them for 4 years, still no issues.
Are you referring to this firmware issue? https://support-en.wd.com/app/answe...-sata-ssds-critical-firmware-update-available

It seems like it only affects the 250-1TB capacities. Mine are all 2TB. Perhaps why I haven't had any issues. Or maybe it's on drives manufactured more recently.

 

[JoanTheSpark]

Wished I had found this thread earlier.. ah well. In Germany they say "From the rain into the eaves" (~futile change).

I had a ZFS-1 pool made up of 3 Silicon Power 1TB SATA 2.5" SSDs for the jails and more responsive datasets for SMB-shares for 12 months.. 2 months ago the first SP drive reported "Device: /dev/adaX, Y Currently unreadable (pending) sectors." This was the moment I ordered 2 spare drives of the same kind - till those arrived the SSD in question got worse and ultimately didn't appear in the system anymore.
Once the spares arrived I replaced the faulty one and 2 weeks later another one of those started with the same issues.. so I put the spare in and as I naturally recognized a trend I opted to buy 3x Crucial MX500 SSDs. It's now been 3 days since one of the SPs started to show that error and today I moved the pool over to the MX500 SSDs. As those had been a tad smaller than the SPs I reconstructed the pool from the snapshots on the spinning HDDs (which is also why all of this isn't really critical, more of a nuisance + waste of money/time).

TL;DR: I might have bet on the wrong horse (Crucial) and need to splurge for Samsung it seems.