Volume balancing doesn't seem to be happening

mrcpu

Cadet
Joined
Mar 10, 2020
Messages
5
36 bay supermicro storage. pool0 is 2 raidz2 6 drive vdevs. Played around, tested things, all appears well. Performance adequate for my needs, everything hunky-dory.

Added 24 more drives as a new pool pool1, which is 2 12drive raidz3's. Ran all the dd and smartctl tests, all good.

So I know the best thing would be to destroy pool0, add it to pool1 and then start copying data on there. But that ain't gonna happen. Files move off and on regularly in large chunks, over the course of a couple months, practically everything will have been deleted and replaced that over time it should balance itself out.

So I started up a zfs send on pool0, wtih a zfs recv on pool1. Some of the datasets are finished copying but I'm noticing the pool isn't balanced or I'm misinterpreting something.

From zpool iostat -v (pool0, nice and balanced, (and yes I know it's full, i just kept cramming data on there to see how bad performance really got.)

The numbers bolded below for pool 1 is what I'm not following. No files are terabyte sized, vast majority are 1-2MB, with maybe a 100 files in the gigabyte range.
pool0 30.8T 1.68T 1.34K 108 41.0M 1.58M
raidz2 15.4T 859G 685 54 20.5M 809K
gptid/852da208-6f25-11ea-acd7-002590c44e80 - - 136 12 4.74M 247K
gptid/8576f0f7-6f25-11ea-acd7-002590c44e80 - - 135 12 4.76M 247K
gptid/85d3fb59-6f25-11ea-acd7-002590c44e80 - - 136 12 4.77M 249K
gptid/85ed610c-6f25-11ea-acd7-002590c44e80 - - 140 12 4.73M 247K
gptid/861687f9-6f25-11ea-acd7-002590c44e80 - - 137 12 4.76M 246K
gptid/85e3f26d-6f25-11ea-acd7-002590c44e80 - - 138 12 4.78M 248K
raidz2 15.4T 861G 686 54 20.5M 809K
gptid/8539145e-6f25-11ea-acd7-002590c44e80 - - 135 12 4.76M 246K
gptid/85233187-6f25-11ea-acd7-002590c44e80 - - 138 12 4.75M 248K
gptid/84fd52c7-6f25-11ea-acd7-002590c44e80 - - 137 12 4.76M 248K
gptid/85530fe6-6f25-11ea-acd7-002590c44e80 - - 140 12 4.76M 247K
gptid/85c81628-6f25-11ea-acd7-002590c44e80 - - 132 12 4.76M 249K
gptid/86003ec6-6f25-11ea-acd7-002590c44e80 - - 137 12 4.77M 248K
-------------------------------------- ----- ----- ----- ----- ----- -----
pool1 9.99T 55.0T 0 5.14K 4 155M
raidz3 6.50T 26.0T 0 3.33K 2 101M
gptid/b0f7542e-c348-11ea-ad9c-002590c44e80 - - 0 512 5 12.8M
gptid/b150d822-c348-11ea-ad9c-002590c44e80 - - 0 531 5 12.8M
gptid/b13ea07f-c348-11ea-ad9c-002590c44e80 - - 0 535 5 12.8M
gptid/b1656077-c348-11ea-ad9c-002590c44e80 - - 0 415 6 9.01M
gptid/b2f76d98-c348-11ea-ad9c-002590c44e80 - - 0 531 5 12.8M
gptid/b42b8913-c348-11ea-ad9c-002590c44e80 - - 0 546 5 12.8M
gptid/b3fcb734-c348-11ea-ad9c-002590c44e80 - - 0 534 5 12.8M
gptid/b3d2fd33-c348-11ea-ad9c-002590c44e80 - - 0 391 6 8.71M
gptid/b3856b69-c348-11ea-ad9c-002590c44e80 - - 0 545 5 12.8M
gptid/b40d7830-c348-11ea-ad9c-002590c44e80 - - 0 542 5 12.8M
gptid/b4613532-c348-11ea-ad9c-002590c44e80 - - 0 534 5 12.8M
gptid/b41c6571-c348-11ea-ad9c-002590c44e80 - - 0 403 6 8.85M
raidz3 3.49T 29.0T 0 1.81K 2 54.0M
gptid/b43a3cd9-c348-11ea-ad9c-002590c44e80 - - 0 339 0 6.91M
gptid/b393222b-c348-11ea-ad9c-002590c44e80 - - 0 337 0 6.91M
gptid/b4bcac80-c348-11ea-ad9c-002590c44e80 - - 0 334 0 6.89M
gptid/b884961c-c348-11ea-ad9c-002590c44e80 - - 0 253 1 5.06M
gptid/b93b3e09-c348-11ea-ad9c-002590c44e80 - - 0 336 0 6.91M
gptid/b9748695-c348-11ea-ad9c-002590c44e80 - - 0 335 0 6.91M
gptid/b995bbc6-c348-11ea-ad9c-002590c44e80 - - 0 336 0 6.89M
gptid/bb1cad53-c348-11ea-ad9c-002590c44e80 - - 0 107 1 4.13M
gptid/bbd97de3-c348-11ea-ad9c-002590c44e80 - - 0 337 0 6.91M
gptid/bc436718-c348-11ea-ad9c-002590c44e80 - - 0 337 0 6.91M
gptid/bc56062f-c348-11ea-ad9c-002590c44e80 - - 0 336 0 6.89M
gptid/bc5fcd50-c348-11ea-ad9c-002590c44e80 - - 0 258 1 5.15M
-------------------------------------- ----- ----- ----- ----- ----- -----
 
Top