Slow ZFS scrub on one of two pools

[Scharbag]

capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
backuptank 10.8T 2.81T 3.09K 0 205M 0
raidz1 10.8T 2.81T 3.09K 0 205M 0
gptid/074d662b-4a15-11e3-8c1b-a0423f00c9d2 - - 1.38K 0 55.9M 0
gptid/08019643-4a15-11e3-8c1b-a0423f00c9d2 - - 1.08K 0 54.2M 0
gptid/08d49d40-4a15-11e3-8c1b-a0423f00c9d2 - - 1.07K 0 55.7M 0
gptid/09b0102b-4a15-11e3-8c1b-a0423f00c9d2 - - 1.15K 0 55.7M 0
gptid/9408ebea-4cb2-11e3-8c1b-a0423f00c9d2 - - 1.17K 0 55.3M 0
cache - - - - - -
gptid/0b0e81a5-4a15-11e3-8c1b-a0423f00c9d2 8.18G 66.3G 0 168 0 20.9M
-------------------------------------- ----- ----- ----- ----- ----- -----
bigtank 12.4T 9.30T 293 0 29.1M 0
raidz2 12.4T 9.30T 293 0 29.1M 0
gptid/e6a73ae3-9970-11e2-85a0-001b21a1f194 - - 187 0 5.82M 0
gptid/dd9964f3-b8f7-11e2-a2f7-a0423f00c9d2 - - 251 0 8.98M 0
gptid/e7c2be90-9970-11e2-85a0-001b21a1f194 - - 170 0 5.84M 0
gptid/e852f0cb-9970-11e2-85a0-001b21a1f194 - - 154 0 5.39M 0
gptid/5d6bf0f5-3379-11e3-a53d-a0423f00c9d2 - - 197 0 6.58M 0
gptid/e970323a-9970-11e2-85a0-001b21a1f194 - - 171 0 5.66M 0
gptid/ea1386cd-9970-11e2-85a0-001b21a1f194 - - 182 0 5.62M 0
gptid/eaa25e83-9970-11e2-85a0-001b21a1f194 - - 220 0 7.91M 0
cache - - - - - -
gptid/69e63616-56ee-11e3-8c1b-a0423f00c9d2 10.9G 63.7G 0 150 0 18.5M

-------------------------------------- ----- ----- ----- ----- ----- -----

Above is at the beginning of a scrub. Below is after a while of scrubbing.




capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
backuptank 10.8T 2.81T 4.42K 0 552M 0
raidz1 10.8T 2.81T 4.42K 0 552M 0
gptid/074d662b-4a15-11e3-8c1b-a0423f00c9d2 - - 2.04K 0 142M 0
gptid/08019643-4a15-11e3-8c1b-a0423f00c9d2 - - 2.68K 0 145M 0
gptid/08d49d40-4a15-11e3-8c1b-a0423f00c9d2 - - 2.39K 0 142M 0
gptid/09b0102b-4a15-11e3-8c1b-a0423f00c9d2 - - 2.30K 0 142M 0
gptid/9408ebea-4cb2-11e3-8c1b-a0423f00c9d2 - - 2.08K 0 142M 0
cache - - - - - -
gptid/0b0e81a5-4a15-11e3-8c1b-a0423f00c9d2 9.81G 64.7G 0 0 0 0
-------------------------------------- ----- ----- ----- ----- ----- -----
bigtank 12.4T 9.30T 1.39K 0 157M 0
raidz2 12.4T 9.30T 1.39K 0 157M 0
gptid/e6a73ae3-9970-11e2-85a0-001b21a1f194 - - 579 0 28.4M 0
gptid/dd9964f3-b8f7-11e2-a2f7-a0423f00c9d2 - - 591 0 27.5M 0
gptid/e7c2be90-9970-11e2-85a0-001b21a1f194 - - 543 0 28.7M 0
gptid/e852f0cb-9970-11e2-85a0-001b21a1f194 - - 553 0 28.5M 0
gptid/5d6bf0f5-3379-11e3-a53d-a0423f00c9d2 - - 641 0 28.6M 0
gptid/e970323a-9970-11e2-85a0-001b21a1f194 - - 548 0 27.7M 0
gptid/ea1386cd-9970-11e2-85a0-001b21a1f194 - - 566 0 27.6M 0
gptid/eaa25e83-9970-11e2-85a0-001b21a1f194 - - 629 0 28.2M 0
cache - - - - - -
gptid/69e63616-56ee-11e3-8c1b-a0423f00c9d2 12.4G 62.1G 0 0 0 0
-------------------------------------- ----- ----- ----- ----- ----- -----

I am not really sure why my RaidZ2 pool takes 21 hours to scrub while my RaidZ1 pool takes about 5. I just updated to 9.2 and still seem to have the same issue.
As you can see above, it seems that the Z1 pool allows far more operations than the Z2 pool. Is this a setting that can be tweaked. Watching

Code:

zpool iostat -v 1

backs this up as I often see 3-5K operations on the Z1 pool whereas the Z2 pool rarely gets above 2.7K.
Any suggestions would be awesome.
Thank you,

 

[cyberjock]

There's ALOT that goes into pool scrubs including zpool history since creation and the quantity of data stored. I have 2 pools and my RAIDZ2 scrubs about twice as long as my RAIDZ3 despite my RAIDZ3 being more than double the size, triple the disks, and about four times the throughput.

It's something that you should just take it for what its worth. Now if it goes from taking 8 hours to 20 hours and you didn't add alot of data then you should start trying to figure out if a disk is bad. But just posting numbers and saying that X is slower than Y is no comparison at all. That's apples-to-oranges.

 

[Scharbag]

Yeah, I hear ya that there are a lot of things going on with a scrub. Data is similar but the big tank does have a few directories that have 1000's of tiny files. From my experience, these are a real PITA for any file system. That may influence overall speeds.

Regardless, FreeNAS is pretty pimp and ZFS is totally cool.

 

[titan_rw]

Scrub times also depend on how fragmented the pool is.

I had a ~20tb pool (~10tb used) that went from 8.5 hours to scrub to 3.5 hours simply by removing 800 gigs of data. It's just that the 800 gigs of data was highly fragmented within the pool. So scrubbing it incurred a lot more head seeking.

 

[cyberjock]

Scrub times also depend on how fragmented the pool is.

I had a ~20tb pool (~10tb used) that went from 8.5 hours to scrub to 3.5 hours simply by removing 800 gigs of data. It's just that the 800 gigs of data was highly fragmented within the pool. So scrubbing it incurred a lot more head seeking.

aka "pool history" which I mentioned :D

Quite literally, zpool scrubs are a function of your hardware and every write your pool has ever had done to it.

 

[titan_rw]

aka "pool history" which I mentioned :D

Sorry. When you said 'zpool history' I thought of the actual output of 'zpool history tank'. And not the write history of the pool.