Full Zpool and expansion questions.

[climb2bhi]

My zpool is at 7.3 TB out of 10.2 usable. (18TB physical via 6x3TB RAIDZ2) And I have much more than the available 2.9TB of data that I would eventually like to put on my system. So obviously a expansion will be made this Winter via another 6x3TB Vdev to my zpool.

My questions are should I stop adding data now while the first Vdev is not at 99% to prevent problems and help things like scrubs, and prevent unhappy hard drives? And secondly when I add the new Vdev to the zpool will the data "equalize" creating more free space on the first Vdev and automatically filling 50% of it to the new Vdev? If not how full should I go on Vdev1 before I stop adding data, save money, and add Vdev2.

Thanks.

 

[cyberjock]

It will not move data from one vdev to the other, but when you start adding data ZFS will preferentially put it on the new vdev.

I wouldn't fill the pool beyond 80%. So in your case I'd fill it to 80%, then wait until you add the new vdev, then add more data. ;)

 

[jgreco]

Also, consider carefully whether or not 3TB drives are the best choice for expansion.

 

[climb2bhi]

Thanks Cyberjock.

This raises new questions:

Is there any advantage to equalizing the Vdevs manually, once I have two, by removing small amounts of data and putting it back on? Obviously prior to the new larger 2 Vdev Zpool approaching 80%. (Data was put on mostly in alphabetical order (so < "M" will mostly be on Vdev1), I have fast (105MB) transfer speeds, and all data is backed up on optical discs)

Also the Web GUI does not seem to have any way to track what percentage of data is on a Vdev. Web GUI > Storage, and Reporting > Partition, show how full the Zpool is. Is there a shell command that will show what percentage is used on a Vdev or individual hard drives once there are two Vdevs? Or does the Web GUI display change after there is a 2nd Vdev?

Thanks for answering the questions of a noob. I'm trying to keep my system as happy as possible.

Climb

 

[climb2bhi]

jgreco,

I am considering 4TB Drives. Price and savings will probably dictate this year.

 

[cyberjock]

Is there any advantage to equalizing the Vdevs manually, once I have two, by removing small amounts of data and putting it back on? Obviously prior to the new larger 2 Vdev Zpool approaching 80%. (Data was put on mostly in alphabetical order (so < "M" will mostly be on Vdev1), I have fast (105MB) transfer speeds, and all data is backed up on optical discs)

Well, this is a sore subject with me so pardon me if I sound rude, etc.

Technically, you *could* see benefits if you were running 10Gb LAN, were in a situation where you were trying to demand more than 100% I/O at all times, etc. But the reality is that your 1Gb LAN is going to be your bottleneck. So it's not really going to matter. Of course, every person that asks that question and I tell them not to do it because of all the hassle, risk of screwing things up, etc do it anyway and then try to argue that it's so much faster. Sorry but if you were bottlenecked at your LAN before it's still going to be bottlenecked at your LAN.

There is no way to identify vdev usage as far as I know. If you want to printout all of the metaslabs that are free in your pool and trace them to vdevs you could figure it out. But it's one of those things where you have to ask yourself why you are adding vdevs. If it is because of performance you're in a position where you are doing lots of random I/O workload, and the new writes are going to quickly even things out automagically. If it's because of disk space then you already aren't bottlenecked and so there's no reason to worry.

 

[jgreco]

There are some potential benefits to trying to even it out, like less pain/better performance during scrubs, but if you hadn't already identified that as an issue, you're not too likely to notice. I'd tend to agree with cyberjock - just let ZFS do its thing and as the new space fills, it all works out.

 

[Dennis.kulmosen]

If you really want even it out a bit, you could slowly replace your excisting 3 TB with 4TB. That way your used percentage will be lower and after that you expand your pool with old 3TB drives, that way your vdevs will be more even without copying data back and forth. You can replace the old drives while they are still plugged in, thereby maintaining redundancy in case things should go bad.
But as the others says is it really worth the hazzle??


Sent from my iPhone using Tapatalk

 

[jgreco]

"vdevs will be more even" is a pointlessism. If you replace the 3's with 4's, you will still end up with disks where the metaslabs in the front 75% of the disk are overly full and the remainder of the disk is empty. What you want is for the data to be evenly distributed amongst the metaslabs, but that can only happen if you move data.

You need to remember that discussions of loading on vdevs involves both spindle (IOPS) and space allocation factors, among other things, and usually when discussing the space allocation factor, we're really implicitly assuming that the data has been spread evenly over the vdev, "cuz that's how vdevs work." Except they don't work that way when you do a replace/resilver expansion cycle.

 

[climb2bhi]

Thanks everyone,

I was not worried about performance on I/O just wanted things like scrubs to be happy. So it was a space concern. Which I will deal with in the future.

This year (for money reasons) the second Vdev will be 6x3TB. And the following year, after more hours on what are now fairly low hour drives on Vdev1, those drives will be swapped to 6x4TB. And a year or two later Vdev2 goes to 6x4TB.

Cyberjock, I will heed your 80% warning and stop at around 75%, and expand shortly after then continue on. That information is not in your excellent guide under expansion. So maybe it could be added as a note next time you update your guide. Even better would be FreeNAS warning you when things reach 80%ish.

It is good to learn more about how the data is spread out, how things work, and what can and can't be done with monitoring.

Thanks again.

 

[danb35]

Even better would be FreeNAS warning you when things reach 80%ish.

It already does.

 

[climb2bhi]

Perfect.

 

[eddie200112]

Hi looking to expand my zpool as well. Any good step by step guides that I can trust? My pool has 2 vdevs each with 6x4TB drives. I will be replacing 1 vdev now with 8TB drives and the other a year down the road perhaps.

 

[danb35]

Did you look at the manual before you necro'd a two-year-old thread? It has pretty detailed instructions on replacing disks to expand a pool.