Looking for update suggestions for NAS with aging hard drives

[Toadlips]

Hi all,

Looking for some direction on how to proceed with upgrading my existing NAS. Here's what I currently have:
- 6x3TB RAIDZ2, 8.7 year old WD Red Drives with 77,000 hours on them. SMART tests pass for all drives. One drive has a single reallocated Sector.
- SuperMicro X10SLL-F mobo (6 SATA ports)
- Xeon E3-1281 v3
- 91% utilization on my pool
- 32GB RAM
- M1015 - only 2 ports in use for some spare drives, so full 8 ports could be utilized
- Case has 6 official HD slots, plus I added a 3x3.5 to 2x5.25 bay to host my misc drives. I could potentially add another 3x3.5 to 2x5.25 bay adapter to host a total of 12 drives in the case.

I use my NAS for Plex, serving files, backup, and, more recently, PhotoPrism. As far as I can tell, the existing hardware meets my needs. However, it is clear that at the very least I need to get some more storage, and I'm concerned about my aging hdds.

Would the general consensus be to continue to use the aging array and just let RAIDZ2 do its job as the drives naturally fail? I was thinking I should replace the drives with 6x8TB, but it pains me to let go of the 6x3TB that are still functioning. I'd really like 24 to 40TB of storage in the end, but I'm curious what you guys think would be the best way to get there. Should I keep the 6x3TB and add another pool of 6x6TB? Ditch the 6x3TB entirely and replace with new 6x8TB? I hate to waste the old drives, but I also think the energy costs/heat/vibration might make it silly to run 12 drives in a mid tower case.

Also, if there are some cheap options to improve my cpu/mobo/ram in a way that would significantly boost performance and allow me to run SCALE, it might be worth it to me, although I don't have strong use case for virtualization at this point. I'm open to used hardware on eBay, etc.

I don't want to derail the discussion further, but another consideration is that I need to upgrade my current desktop machine for gaming and some productivity. No, no - it's not for me. It's for the children. They need an upgrade, so I'm considering doing them a favor by upgrading my machine so that they can have my old processor. I will do anything for my children. I saw that the W680 can support ECC RAM, and I had some wild thoughts last night that I would get that ASUS W680-ACE, and somehow virtualize TrueNAS on it, stuff my old 6x3TB drives in there, and use it as my Windows gaming and productivity workstation. You know -- I want everything in one box. Please tell me I'm crazy so that I abandon this dream, but it would be pretty cool if I could somehow virtualize TrueNAS and use those 6x3TB as backup, while running Windows with my video card and playing games. Is that even possible? If it is, please don't tell me.

Thank you for your time and any suggestions you may have!

Sincerely,
Toadlips

 

[chuck32]

Im on mobile so just some pointers:

- 91% utilization on my pool

That's too high, you need to add space or lose some files. Stay under 80 %.

Also, if there are some cheap options to improve my cpu/mobo/ram in a way that would significantly boost performance and allow me to run SCALE, it might be worth it to me, although I don't have strong use case for virtualization at this point. I'm open to used hardware on eBay, etc.

Why wouldn't it run scale? I run scale on my replication system in my signature. I don't use VMs there though, but my CPU isn't that capable.

ould the general consensus be to continue to use the aging array and just let RAIDZ2 do its job as the drives naturally fail? I was thinking I should replace the drives with 6x8TB, but it pains me to let go of the 6x3TB that are still functioning.

Do you have backups?

I'd let the drives run, but because you're out of storage I'd recommend you upgrade. If you go 6 or 8 drives is up to your needs, but remember that you cannot expand a raidzx currently.

 

[NugentS]

The CPU 4c/8t @ 3.7GHz is great. Its only issues are:
1. Lack of PCIe lanes (16) as its not a server chip
2. No iGPU for transcoding
3. Max memory is 32GB

It will still run scale if you want, but if Core works for you - stick with that. However I am not sure if Core does Nvidia transcoding. If transcoding is something you want then Scale and an Nvidia eGPU would seem sensible. Do you have the slot for a eGPU? The board has a 2 3*8 and 1 2*4 so you could run a GPU in the second 3*8 slot which would work well at transcoding (for the right GPU)

Do you have a proper backup?
If yes - then keep using the existing disks - but buy a couple of larger cold spares so when one fails you replace with a larger and start the upgrade
If No - then start replacing the disks with larger (best bang for the buck) disks using the replace function. Do not diconnect a disk first, simply replace an existing disk - that way you maintain parity levels.
As a matter of paranoia, make sure you get a backup of critical data first

At 91% however - start replacing disks.

 

[Constantin]

If budget is a consideration, then I’d look into replacing those 3TB drives one by one with He8 or He10s from goharddrive.com. 5year warranty (from goharddrive) $80, low power consumption / low heat. Then your pool grows 3x and should feel young again. Meanwhile, nothing else has to change.

Now, you can upgrade this pool and then build a more performant machine with new toys like L2ARC, etc but that’s a separate story. I second having pre-qualified cold spare(s) ready for replacing your HDDs, regardless of age. Pre-qualified = tortured at length with bad blocks or eqv to ensure the drive can handle resilvering and life inside your NAS. Good luck.

 

[rvassar]

You probably need to decouple the upgrade from the pool rework. 77k hours is past the sell by date for almost all HDD's I can think of. Even legendary HGST He's become trouble eventually. Drives all the same age, they may all decide to fail together as one cooks the others via heat of a bearing failure, etc... I wouldn't consider those drives for even secondary use.

Make sure you have a good backup. Start replacing drives in-place and get your pool back to a safe condition with younger drives and < 80% utilization. Then plan your upgrade. This could even be cheap used 4Tb CMR drives while you plan a more extensive re-work. The 4Tb drives could then become your kid's entertainment/gaming drives.

 

[Constantin]

Agree re having a backup but after that, what is the risk in incrementally replacing every 3TB drive with a 8TB+ drive and letting the pool resilver 100% between each swap?

If the pool replacement / resilver approach fails, simply replace all drives and restore from backup. In the meantime, still have access to the data, etc.

 

[rvassar]

Agree re having a backup but after that, what is the risk in incrementally replacing every 3TB drive with a 8TB+ drive and letting the pool resilver 100% between each swap?

If the pool replacement / resilver approach fails, simply replace all drives and restore from backup. In the meantime, still have access to the data, etc.

The risk would be the same. My reply was more a call to expediency, don't wait to re-plan the build, and to dissuade re-utilization of 77k hour HDD's for any reasonable use. If the cost of the 4x 8Tb drives limits the OP's rate of action, then it's reasonable to consider smaller/cheaper options with an option to re-use those for the other proposed purpose at a later date.

I have a couple non-He 55k+ hour drives laying around that I use for solo tests & one-off data transfers. Generally, I don't trust any data placed on them to survive a power cycle.

 

[Constantin]

The He10 drives in my pool are approaching 40khrs so I’m starting to build a whole replacement set. Most of the drives I bought in 2018 as part of the NAS upgrade have held up great, only a few had to be returned to goharddrive. The few that required warranty exchange / refund (5-years, from goharddrive) were taken w/o issue.

My plan is to prequalify a whole stack, then set aside for inevitable failures. With a Z3, I expect plenty of opportunity for resilver instead of pool loss. Cost of used He10s w/a 5-year warranty at goHDD is only $8/TB, that’s pretty hard to beat. After all, we’re only one flood away from the entire HDD supply chain from going bezerk, again.

 

[Toadlips]

Thank you all for your advice and recommendations, chuck32, NugentS, rvassar & Contantin! I appreciate it!

The overwhelming theme I'm picking up on: get a good backup because the drives are on borrowed time. As fate would have it, I tried to backup my pool to an external drive and ran out of space on my backup drive. :) I'm about a month shy of having all of my photos backed up, so that's priority #1.

Nobody seems to think I should keep the old hard drives, and that makes sense to me. The one thing I neglected to mention is that I'm running legacy encryption on the drives, and that's already preventing a potential move to Scale since my pool needs to be upgraded. I'm thinking I'll have to have both pools up at the same time and transfer the 6x3TB to my new 6x8TB pool...then I can retire the 6x3TB. I should be able to manage that for the time needed to burn in the drives and transfer.

I'm pleasantly surprised that no one has suggested that I need to dump my other hardware. It's getting a little long in the tooth, but it seems to be holding up pretty well for my current needs. If I do indeed take the plunge into SCALE, I'll have to see if I can benefit from an eGPU...but I'm thinking that's some time down the road and probably will be when I get some beefier hardware. I'm hoping to roll with what I have for another few years.

Regarding the hard drive purchase, it would be tough for me to try the used drives. I imagine those drives have been ridden hard and put away wet for 5 years straight! Then again, participation in my pool would probably feel like a walk in a cool meadow for such drives! :) I think I'm gonna stick with the drives that got me this far, the WD Red Plus 8TB.

Here's the scoop, best deal I can find is from WD's own site. I can get 8TB drives with the 128MB cache for $129.99ea:
https://www.westerndigital.com/products/internal-drives/wd-red-plus-sata-3-5-hdd?sku=WD120EFBX

What do you all have to say about the 128MB cache rather than 256MB? Bear in mind that my 1Gbps connection to the NAS hasn't caused me much pain, but that means my network connection is the current bottleneck. I have no idea what the limited cache would mean in terms of performance so I'm curious what the consensus is on that.

Here's a price comparison from the WD website, with the various capacities and cache options:

Looks like 6x8TB with 128MB cache would yield about 32GB in RAIDZ2, at a cost of $24.37 per TB. I think that's the sweet spot. Another possibility would be 8x4TB...lower total capacity, but reasonable price and double my current capacity. If anyone has any thoughts on this, in terms of reliability, performance, etc. of more drives vs. fewer drives, more cache, less cache, more platters, less platters etc. I am all ears!

And, not to beat a dead horse, but after some research it seems I'm not the only one who would like to run a NAS/Windows gaming machine on the same box. However, that looks like an uphill battle at the moment even though it seems that it all should be possible with existing technology. I'll wait on that till things settle down on that front! Still, I'm finding the state of memory ECC support for consumer hardware to be very much lacking.

Everyone says "if you want ECC, get an AMD cpu", but when it comes down to finding a motherboard that supports ECC with mainstream cpus that explicitly support ECC (i.e. 7900x), I have yet to find a regular ATX board. So far, I've just seen gaming mobos and one 1U board that does support ECC but isn't in a standard form factor. I have more research to do on that front because I'm baffled these things aren't more readily availble. My thought process is that I'm more than willing to take a 1 to 2% loss in memory performance in order to ensure system stability and, in the future, maybe 5 years from now, I could repurpose that same mobo/cpu/ram in a new TrueNAS SCALE installation. Is that so much to ask? :)

Thanks all!

 

[danb35]

Everyone says "if you want ECC, get an AMD cpu"

I don't think I've ever seen such a recommendation, much less as a majority opinion, though I don't doubt some are saying it. The standard recommendation here is to use decent server-grade kit, and don't worry about (indeed, probably look for) used gear a generation or two old. The result of that will most often be Intel rather than AMD.

I'd agree there isn't an immediate reason to replace the motherboard/CPU, unless you find the 32 GB max RAM capacity to be too limiting. In that case, you could look for an X10-generation Xeon E5 board, or a X11-generation Xeon Scalable board. Both are widely available on the secondary market, fairly inexpensive, and will support hundreds of GB of RAM.

 

[Toadlips]

Everyone says "if you want ECC, get an AMD cpu"

I don't think I've ever seen such a recommendation, much less as a majority opinion, though I don't doubt some are saying it. The standard recommendation here is to use decent server-grade kit, and don't worry about (indeed, probably look for) used gear a generation or two old. The result of that will most often be Intel rather than AMD.

Sorry, I should have provided some context on that! You are correct, nobody here is saying that. I've been looking at ECC support for current consumer CPUs, like Intel's i7-13700K and AMD's 7900X and digging through forums on Reddit, etc., where people are discussing ECC support for the W680 chipset. There is a lot of murkiness surrounding that and questions about whether or not ECC is "truly" supported. In those forums, I see a lot of people saying "just get an AMD cpu if you want ECC", but I'm finding it to be just as difficult to find AMD consumer boards that support ECC and as many questions.

In a perfect world, I'd like to get a workstation board, like the Asus Ws W680-ACE, and potentially use it for a TrueNAS installation somewhere down the line after I'm done using it for my Windows general purpose/gaming/productivity machine. Might be better to stick with tried & true for the NAS build and not get too greedy. Ultimately, my general purpose machine is due for an update and a 7900X or i7-13700K would be great, and if I could use ECC RAM with it and repurpose it for TrueNAS somewhere down the road, it would be the best of both worlds. That's my thought process.

I appreciate the suggestions on current, reasonable hardware. I plan on doing some more stuff with my photo library (PhotoPrism possibly), and if I bump into obstacles, I may find myself in need of an upgrade!