Striped mirrors (RAID10) - can you increase capacity of disks already in the pool?

[ProblematicLlama]

Hi,
TL;DR: I know you can easily add disks to a striped mirror to increase capacity, however... is it possible to replace 2x mirrored 3TB disks in that setup with 2x 8TB disks and get the additional capacity? Or is the only way to increase capacity to add another mirror to the end of the stripe?

I have recently started to develop an issue with one of my disks which is in a raidz1 array. I'm considering eventually moving to something else (either raidz2 or striped mirrors) as resilvering is pretty nervewracking in raidz1.

Right now I have 5x 3TB WD Reds but I'm thinking if I do go ahead and change everything, I'll gradually get 8TB drives - partially so I can actually grab the data off the existing setup I have ready to copy over to the new one! The reason I'm leaning more to striped mirrors is that I have heard that replacing a disk with raidz2 takes a *very* long time, so just imagine that when I eventually have a load of 8TB drives.. it would be nuts. Also it will be quite a while before I can afford to make them all 8TB disks (and obviously ever disk has to be the same size in order to get the full capacity in z1-3).

Anyway, at first, the array would probably look like this:

6x3TB WD Red Drives = 9TB in a striped mirror (obviously 3 mirrors striped, and beforehand I'd copy as much data as I can onto 8TB disks)

Then I'd add in the 2x 8TB disks.

So basically - is that possible or is the only way to increase capacity to add disks at the end of a stripe (or, I guess, dangerously seaparate the mirror, create a new unmirrored stripe with the new capacity, copy the data on the original unmirrored stripe over to the new one, then retroactively mirror them... if that's even possible.. and that sounds incredibly dangerous!)

If it is possible, how exactly does FreeNAS do it? Trying to wrap my head around that part and can only come up with pretty unconvincing methods, e.g. it creates a new vdev on the back half of the disk and adds that to the stripe, but that would be stupid as it would have double striping on the same disk with presumably quite a large performance impact. I guess the sensible way for it to do it would be for it to push all the data from the subsequent disks back - presumably that would be pretty slow and intensive but it would.. probably.. work.

Thanks in advance!

 

[SweetAndLow]

Yes you can expand your pool by swapping out your disks for larger disks. You replace them one at a time then offline the old drive. If you have a spare sata port that is best.

Sent from my Nexus 5X using Tapatalk

 

[ProblematicLlama]

Yes you can expand your pool by swapping out your disks for larger disks. You replace them one at a time then offline the old drive. If you have a spare sata port that is best.

Sent from my Nexus 5X using Tapatalk

Thanks, do you know how it actually achieves the expansion?

 

[SweetAndLow]

You resilver the new drive. It's the same as if a disk goes bad.

Sent from my Nexus 5X using Tapatalk

 

[joeschmuck]

Thanks, do you know how it actually achieves the expansion?

Once you have replaced all the disks the available storage will be expanded. It will not expand a little bit as each drive is added. It's the way it must be done due to the striping of data across all drives.

 

[Jurgen Segaert]

The manual has some great step-by-step instructions on how to do this.

http://doc.freenas.org/9.3/freenas_storage.html#replacing-drives-to-grow-a-zfs-pool

 

[fta]

Once you have replaced all the disks the available storage will be expanded. It will not expand a little bit as each drive is added. It's the way it must be done due to the striping of data across all drives.

You probably meant this, but just for clarification: in a pool of striped mirrors, you only have to replace the two drives in a single mirror to get increased capacity.

 

[nojohnny101]

You probably meant this, but just for clarification: in a pool of striped mirrors, you only have to replace the two drives in a single mirror to get increased capacity.

yes.

 

[ProblematicLlama]

Awesome, thanks for the answers (and the clarification was helpful)

 

[Stux]

Do you actually have to offline the drive after replacing it? Doesn't it get removed from the vdev automatically after you replace it?

I recall from when I tested it, that when you replace a mirror disk, it converts the mirror to a 3way, with the new member resilvering, and when it's done it removes the original disk from the vdev, at least that's my recollection

Then you replace the other disk, and as long as the pool is set to auto expand, it does.

If it's not auto expand, then you can force the expansion

 

[SweetAndLow]

Do you actually have to offline the drive after replacing it? Doesn't it get removed from the vdev automatically after you replace it?

I recall from when I tested it, that when you replace a mirror disk, it converts the mirror to a 3way, with the new member resilvering, and when it's done it removes the original disk from the vdev, at least that's my recollection

Then you replace the other disk, and as long as the pool is set to auto expand, it does.

If it's not auto expand, then you can force the expansion

It might offline for you I'm not sure.

Sent from my Nexus 5X using Tapatalk

 

[danb35]

Do you actually have to offline the drive after replacing it? Doesn't it get removed from the vdev automatically after you replace it?

At least in RAIDZn vdevs, the old disk is automatically removed when the replacement finishes. I can't see any reason that mirrors should be any different, but I haven't tried.

 

[SFoskett]

This is the main reason I recommend striped mirrors instead of RAIDZ for many-disk setups. It's much more friendly to be able to replace two disks at a time and grow a pool than have to replace every drive in a RAIDZ (usually many more than two!) to add space. Plus, performance is great. The downside is less usable capacity and reduced MTDL and risk if you lose a disk, but I'm usually willing to bear that.

 

[Spearfoot]

This is the main reason I recommend striped mirrors instead of RAIDZ for many-disk setups. It's much more friendly to be able to replace two disks at a time and grow a pool than have to replace every drive in a RAIDZ (usually many more than two!) to add space. Plus, performance is great. The downside is less usable capacity and reduced MTDL and risk if you lose a disk, but I'm usually willing to bear that.

Yeah, I agonized over pool layout when I built my all-in-one last year. Tried both mirrors and RAIDZ2 and ended up staying with RAIDZ2 because, in my particular case, the performance difference was negligible, RAIDZ2 is safer than 2-way mirrors, and I got more usable capacity. Plus I could use all 8 of my SAS ports - 7 x HDDs in RAIDZ2 w/ Intel S3700 SLOG device. Don't want to waste a perfectly good SAS port! :)

 

[SFoskett]

People do worry about performance a lot, but that's not really an issue for most FreeNAS users I think. Unless you're pounding on the disks with random I/O (iSCSI/VMware) I can't believe you'll really notice the performance difference of RAID1 over RAIDZ. Especially since most FreeNAS still has a single 1 GbE port as a front-end!

In my case, I have a RAID1 pool using a pair of enterprise SAS SSD's and a RAIDZ pool using green HDDs. They perform exactly the same when accessed as a file server from SMB and AFP clients. Because that protocol and GbE is the bottleneck, not anything inside the box.

The same is true of compression. My Xeon E3 starts breathing heavy when processing gzip level 6 data but it never gets past 70% utilization so it's of no concern. Of course the space savings are small so I don't bother with gzip except for "deep archive" data.