Stripe of mirrors expansion

[viniciusferrao]

Hello guys, I've a question about the expansion of a stripe of mirrored vdevs.

Let's imagine 4 disk in a zpool. 2 mirrored vdevs inside. When I wrote, 1TB of data, it's get stripped through the two vdevs resulting in 500GB in each disks, right?

Let's expand the pool now, put more 4 disk and let's write again 1TB of data. What happens now? The initial 500GB data in each disk reorganizes itself or the new data write 250GB in each disk resulting 750GB used in the first 4 disks and 250GB in the last 4 disk?

If the second scenario happens, what can we do to optimize this?

Thanks in advance,

 

[cyberjock]

First, don't bother trying to reallocate your data. While you'd expect to see a performance increase, practical application has shown this does not matter. ZFS seems to do stuff to keep speeds up.

Second, when you add a new vdev, that vdev will be somewhat preferred. It's MUCH more complex than saying "it'll equal out all data". There is no way to explain easily how it predicatively stores your data. And the new flags feature has histograms that optimize things too.

In short, don't worry about this stuff. Add your vdevs and enjoy the space. The performance will come as data is naturally moved around through the course of writing data to the pool. You are welcome to do a boatload of copying and pasting, but then you're just forcing the new data onto the new vdev, so you really aren't doing yourself a favor at all.

 

[mpfusion]

And the new flags feature has histograms that optimize things too.

I tried to find information about the “spacemap_histogram” feature, but didn't really find anything related to FreeNAS. There's one post on the forum mentioning

right now there is no documentation on this, but I think the metaslab allocator doesn't use this information by default yet (it seems you need to set vfs.zfs.metaslab.weight_factor_enable to enable that, but I may be wrong

Can you enlighten me what the histogram feature does, if it's working out of the box in freenas or if it requires the mentioned tweak and how it “optimizes things”?

 

[cyberjock]

Not too sure on it because there's little documentation on it. I just haven't gotten into the nitty gritty of it since I'm not on 9.2.0 yet.

 

[Dusan]

I tried to find information about the “spacemap_histogram” feature, but didn't really find anything related to FreeNAS.

This is the definitive documentation :): https://github.com/trueos/trueos/commit/33f9d57c4e0599bcddac47a36fd06f3850758491
Some additional information is here:
https://www.illumos.org/issues/4102
http://permalink.gmane.org/gmane.os.illumos.zfs/2570
The mailing list post mentions that the allocation algorithm change is currently disabled.
I did not yet look into it in a detail, but the new algorithm seems to be here (the code comment provides additional information on what it does): https://github.com/trueos/trueos/bl.../uts/common/fs/zfs/metaslab.c?source=cc#L1155
Here it is decided if it is actually used: https://github.com/trueos/trueos/bl.../uts/common/fs/zfs/metaslab.c?source=cc#L1255
And the decision depends on this sysctl: https://github.com/trueos/trueos/bl...s/uts/common/fs/zfs/metaslab.c?source=cc#L212

 

[cyberjock]

I figured Dusan would have the answers. :)

 

[viniciusferrao]

Wooha, a lot of documentation. Thanks dudes!!!

Let-me ask one more thing. When I create a mirrored zpool and them expand it, it use non conventional names. I know it does not make any difference but theres a way to show correctly gptid's?

This is what is happening:

Code:

  pool: test0
 state: ONLINE
  scan: none requested
config:
 
NAME                                            STATE     READ WRITE CKSUM
test0                                           ONLINE       0     0     0
 mirror-0                                      ONLINE       0     0     0
   gptid/8157e976-794f-11e3-a1c0-00259076a639  ONLINE       0     0     0
   gptid/81e86665-794f-11e3-a1c0-00259076a639  ONLINE       0     0     0
 mirror-1                                      ONLINE       0     0     0
   da22p2.nop                                  ONLINE       0     0     0
   da23p2.nop                                  ONLINE       0     0     0

 

[mpfusion]

Thanks for digging into the source. Apparently there's not much documentation apart from source code comments. But that sheds light on how it actually works.

As far as I understand it, the “spacemap_histogram” feature is not activated in FreeNAS and does not optimize anything unless “vfs.zfs.metaslab.weight_factor_enable” is turned on, which is not the case by default on FreeNAS. The question is why this is not activated, since apparently it makes sense. Because it is new/untested? Does it make sense to activate it or is it not recommended at this point?

 

[Dusan]

The question is why this is not activated, since apparently it makes sense. Because it is new/untested? Does it make sense to activate it or is it not recommended at this point?

Did you read the mailing list post I linked above?:

> >>The allocation algorithm that uses this information is
> >>disabled by default and can be enabled via a tunable. It will become
> >>the default allocator once George is satisfied with the amount of
> >>performance testing it has received.

 

[mpfusion]

Sorry, I missed that part.

 

[cyberjock]

How did you expand the pool? Did you use the CLI?

 

[viniciusferrao]

How did you expand the pool? Did you use the CLI?

I've used the GUI. Version 8.3.1-p2.

 

[Dusan]

I've used the GUI. Version 8.3.1-p2.

You should really upgrade to a newer version: https://bugs.freenas.org/issues/1717
FreeNAS 9.x doesn't do gnop tricks to force 4k sectors.

 

[viniciusferrao]

You should really upgrade to a newer version: https://bugs.freenas.org/issues/1717

This machine is under production... I really would like to upgrade, but I don't know if it's safe and which is the best path to upgrade.

Thanks in advance,

 

[Dusan]

Did you try a reboot? (Or to export/import the pool?)

 

[viniciusferrao]

Did you try a reboot? (Or to export/import the pool?)

I can't reboot :)
There are VM's through iSCSI running on this machine! If I will need a reboot, perhaps upgrading would be a better solution... Since I will need to reboot it anyway, right?

 

[Dusan]

I think the .nops will disappear after a boot. FreeNAS can't yank the gnop providers (it used them to force 4k sectors) from under an active pool.
And yes, you need to reboot to upgrade (I think it reboots twice during the upgrade process).

 

[viniciusferrao]

I think the .nops will disappear after a boot. FreeNAS can't yank the gnop providers (it used them to force 4k sectors) from under an active pool.
And yes, you need to reboot to upgrade (I think it reboots twice during the upgrade process).

Any good practice or advice to do the upgrade?

 

[Dusan]

Any good practice or advice to do the upgrade?

• Read the documentation, twice :) (http://doc.freenas.org/index.php/Upgrading_FreeNAS)
• Backup your config DB & encryption keys (if you use encryption)
• If you really want to be sure you know what you are doing test the upgrade in a VM first.
• Do not upgrade your zpool version until you are absolutely positive that you won't need to revert to the previous FreeNAS release.

If you want to minimize the downtime you can also install FreeNAS 9.2 to a new USB stick on a different computer. Then just shutdown your productive system, plug in the new stick, boot and import the config. The added benefit is that if something doesn't work as expected you can swap in the old USB stick. Please note that the root's SSH key is not included in the config backup, so you need to reenter it if you use this method.

 

[cyberjock]

And don't do a zpool upgrade from the command line until you are 100% sure you will NEVER need to go back to 8.3.1. I'm still on v28 and I've been using 9.x for 5 months or so(whenever 9.1.0-RELEASE came out). There's no overly pressing reason to upgrade the pool aside from the warnings. :P