Transitioning a 6x3TB RAIDZ2 vdev to a new layout

[coolnodje]

I've a 6x3TB RAIDZ2 vdev with 3 disks showing "Self-Test Log error count increased " errors on a home server.
Time to replace them!

With that many of them needed to be replaced, and with 4TB being the minimal HDD size with regards to price per TB, I'd like to transition to a 2x 4disks RAIDZ2 vdevs layout.
One with 4x4TB and the other with the 3x3TB still functioning drive + a new 4TB or 3TB if I can still find one.

First off, is it a good idea?
Having multiple RAIDz2 vdev seems to be what is advised on ixsystems blog posts and forum threads.
Now I'm not sure what the difference is between a 4disk and a 6disk RAIDz2 vdev, but it seems to me that I'd be better off with smaller vdev (in term of HDD qty) with regards to the limits of my case and motherboard. More flexibility. (though I don't need all that space and probably won't for the near future I think)

Now in practice, I know that worst case scenario I could do with a 4TB backup for the entire Pool I have.

The ASROCK C2550D4I I'm using has available:
8xSATA3 port
4xSATA2 port

I'm using 6 of the SATA3 for the vdev, +1 for the boot-pool SSD that could well be put on a SATA2.

Maybe my best guess would be to build a new 4x4TB RAIDz2 vdev, plug this on the SATA2 to start with (even though my case can fit 8x3'5 drives only), backup/restore or transfer the existing pool to a new pool on this vdev, then just dismantle the existing 6x3TB VDEV and move the new one to the SATA3 ports.

Would this work as I expect?

Is there a cheaper option?
Buying 3x3TB drives doesn't makes sense. But I could buy 3x4TB and replace the disks within the existing VDEV.
I would lose 1TB per disk, and 1more TB for each new failing disk this until the last of the 3TB disk dies. And I would still have this not very flexible 6disks vdev. Not very appealing .

 

[Constantin]

Can’t speak to the need re replacing drives, though one option is to replace with new stuff and resilver each time.

Whether you need to go a to a multi-VDEV scenario really depends on the use case, which we do not know. For example, I’m perfectly happy with a single VDEV consisting of eight drives in a Z3 but I’m a home user and I don’t have that much data flying around.

Another factor is your network. Unless you put a 10GbE card in the PCIe slot of your motherboard, you may have difficulty benefitting much from multiple VDEVs (bandwidth throttled)

I am not a fan of 4 drive Z2 VDEVs as you lose two drives to parity, so you might as well mirror.

for my use case (long term storage) I’d look instead into replacing the 3tb drives with something bigger (6TB helium?) that will give you room to grow, offer low power consumption and adequate performance.

the benefit of replacing the drives one by one with larger drives is that eventually you have replaced them all and then the VDEV shows the space associated with the bigger drives.

the Z2 configuration is the workhorse of FreeNAS, but that’s not to say your use case may benefit from a different configuration. Tell us more how you use the array and what you want improved. Perhaps then we can help you further.

 

[Pitfrr]

Also, some considerations from a HW perspective.

The AsRock C2550 has two SATA controllers, well actually three...
One intel with 6 SATA connectors and two marvel, a 9172 with 2 SATA and a 9230 with 4 SATA.

Be aware that if you put more than 6 drives you will use the marvel controllers and might run into unstability problems.
At least that is what I experience with my C2750 (running FreeNAS 9.10 maybe it is more stable with the latest FreeNAS release?). I had drives dropping out from time to time, of course each time drives on the marvel controllers. So I had to add an HBA to solve it.
In the beginning it was not such a big deal, since it was my backup server but it became more frequent with time and then it was really annoying.


Regarding the replacement of the drives you could also consider to replace your pool with 6 new drives as @Constantin suggests but keep the 3x 3TB for backup. That could be a nice way to have backups (if not already).

 

[coolnodje]

[...]
the benefit of replacing the drives one by one with larger drives is that eventually you have replaced them all and then the VDEV shows the space associated with the bigger drives.
[...]

Thanks, I wasn't unsure of that particular point.

So you need to wait until ALL the HDD in the vdev are replaced, but then
at least the vdev recognizes the new available space, maybe not automatically, but there's a way to just grow the vdev without having to move the data, correct?

I am not a fan of 4 drive Z2 VDEVs as you lose two drives to parity, so you might as well mirror.

I didn't realize that either, I should do more reading here.

My main use case is pretty much just a home media server and computers backup.
I also experiment using my FreeNAS as a docker server and VM host, iSCSI target, but it really just for fun because FreeNAS offers it. I don't need it and my setup is a bit slow for this.

I find it annoying to have to wait until the 6HDD dies before being able to enjoy the space of the new HDD, that is why I figure smaller VDEVs are more flexible.

And with @Pitfrr warning about my motherboard potential issue (thanks for this!) I should probably stay within the 6 SATA3 offered by the Intel controller.
Maybe it just make sense to do as you suggest, keep the 6disks VDEV replace with 4 or 6TB HDD and patiently wait for all drives to be replaced.

Then what about the latest issue with CMR vs SMR for the Helium disks?
I noticed these Helium have a lot of cache memory, which I find suspicious because this is what SMR requires.
Are these HGST He just as good for NAS as WD RED ?

 

[Yorick]

So you need to wait until ALL the HDD in the vdev are replaced, but then at least the vdev recognizes the new available space,

Correct. It's a pool setting, `autoexpand`. Should be on by default for pools that were created with recent versions of FreeNAS; can be turned on for any pool if it's not on already.

6-wide raidz2 is perfect for a home media server. Can be built with any desired capacity point. I went to 8-wide raidz2 but that's arguably a little nuts.

The HGST He are a little better than the Red, they have a better unrecoverable failure rate. 10E-15 instead of 10E-14, if memory serves. They're all CMR.

The 8TB WD Elements still has HE8 in them, that was the price/capacity sweet spot for me. 10TB is WD Red Air it seems, though the older ones were WD Red He, and 12TB is HGST He again, "probably".

"Shucking" has a price advantage and one disadvantage: You need to take a moment to teach your PSU not to supply 3.3V to the drives, so they'll spin up.

 

[coolnodje]

Sorry, this totally confuses me. I don't understand what you're referring to.

"Shucking" has a price advantage and one disadvantage: You need to take a moment to teach your PSU not to supply 3.3V to the drives, so they'll spin up.

Looking at the official WD website, it seems that they are transitioning away fro their Color marketing, and WD RED are nowhere to be found anymore. Or am I mistaken?
I'm having a hard time figuring out if the equivalent of the WD RED is the HGST Ultrastar DC HC3xx series. They seem to be slightly cheaper for the same capacity.

I'm now a bit worried that I could end up replacing my 3 failing drive with disks that won't be available on the market when my 3 good drives fail. (like if I buy 4TB ones, they may not be available years from now, just like 3TB are not available anymore)

 

[Constantin]

If you can find quality used he8 drives, that is another Option. For example, I found gohardrive.com to be a quality vendor that offers used drives with a 3-5 year gohardrive warranty that I found to be good. (Had to use it on two drives so far). No muss no fuss.

that works best if you are in the same geography / country as the vendor since shipping costs internationally have gone completely insane. 4TB drives are being thrown out of all data centers everywhere so there should be no shortage of supply. A 4 or a 6TB He drive pool could buy you a lot of storage, less heat, less power, etc.

as for the motherboard, I’d plan on it failing unless it was built after March 2017 (or whenever the cutoff date was). Figure out what you want in advance, better to preplan for failure. Buying used (just not anything featuring a c2xxx series CPU) is still a great option.

speaking of planning, I’d always buy an extra disk, burn it in (just like the pool drives) to see if there are any issues, then disconnect and set aside as a cold, but qualified, spare. I have 3.

 

[Constantin]

Lastly, nothing stops you from replacing all drives in one long session - disconnect and replace one drive at a time, allow the pool to resilver, then put in the next one. Read up on this process to not destroy the pool!

This option works best for folk who want to upgrade the array over time (think cash flow) or who have a pretty good idea how quickly their collection is growing and hence start planning in advance for the next disk pool.

So by the time your pool is reaching 80% capacity (which you always want to stay under), the last remaining disks are replaced one by one, allowing the pool to settle back down to 50% capacity or wherever you want to start next.

However, nothing stops you from nuking the pool and starting over, provided you have adequate backup. This is likely the least time-intensive option. With your infrequent experimentation re VMs and so on, I’d consider adding a separate mirrored SSD pool.

 

[Yorick]

and WD RED are nowhere to be found anymore

They're still there: https://www.westerndigital.com/products/internal-drives/wd-red-hdd

Branding changed. WD Red is now "Satan in HDD form factor"; WD Red Plus is what WD Red used to be, as in, perfectly fine for ZFS; and WD Red Pro hasn't changed.

Edit: I realize I just insulted Satan. I am sorry. Of course he's nowhere near as bad as WD Red drives.

 

[Constantin]

Well, at least the color coding checks out re: Lucifer in most of his depictions?

 

[pschatz100]

@coolnodje back to your original question...

I'll throw out the following thoughts:

There is nothing wrong with the idea of having two pools and there is nothing wrong with a 4 disk RaidZ2. One of the benefits of a 4 disks RaidZ2 over two mirrors is that ANY two disks in a RaidZ2 pool can fail before you lose data - whereas you will lose your data if both disks in the same mirrored pair fail. It's not terribly likely, but it is possible. My home system is a 4 disk RaidZ2.

If you originally set up a 6x3TB RaidZ2 pool, then you had 12 TB raw space allocated to data. The least expensive way to go in the near term would be to replace the three failed disks with new disks. If the new disks are larger, the capacity of the pool will remain unchanged until all six disks have been replaced (as was explained in an earlier post.) To me, the main downside here is that I would be worried about the reliability of the three remaining drives. If all six drives are about the same age, and three of them are going bad, then I would not put too much trust in the remaining three drives. But, if you plan to replace all six drive - then you can do the three problem drives now.

If you decided to go with your idea to build a new 4 disk RaidZ2 plus create a 4 disk RaidZ2 by reusing your good 3TB drives plus a new drive, then you are still purchasing 5 drives. At this point, I would suggest staying with the 6 disk pool. However, if you decided to replace the 6 disk pool entirely with a single 4 disk RaidZ2 pool, you would get capacity equivalent to what you have now with four disks 6TB or greater.

 

[coolnodje]

Thanks for your feedback @pschatz100

It comforts me in my decision to simply replace failing HDD with WD RED 4TB (still the sweep spot in €/TB, I calculated I saved 6€ per TB as compared to 6TB disks, and for 3 new drives that's already ~120€)

As I seem to have other worrying issues atm, I felt I'd go for something simple and fast (see https://www.ixsystems.com/community...he-reason-of-unscheduled-system-reboot.87818/).

The He disks option seemed attractive, and I'm glad i know about them now, but next time.

Thank you to all people in this thread, that really helped me.