Scrub slow, Problem or biproduct of having a massive pool?

[Arwen]

One comment about scrubs, (and re-silvers), is that if you have a ton of small files,
they seem to impact the speed to read the entire pool. That appears to be because
it's reading each and every tree, and file with small files causing a branch in the
read stream, (back to the directory entries).

My backup scheme uses RSync of my media server, lap top and desktop, with
multiple copies of the backups, (because I have room now). This seems to cause
the scrubs to slow down, (based on MBps numbers).

 

[DrKK]

Scrub said 325hrs last night, I came in this morning and it was finished.

A little unsettling but at least I can reboot now.

View attachment 10128

Strange, isn't it, how close 737 hours is to the exact scrub time estimate you were given by a pretty experienced FreeNAS guy .... :)

 

[fullspeed]

Strange, isn't it, how close 737 hours is to the exact scrub time estimate you were given by a pretty experienced FreeNAS guy .... :)

Hah well done, i'll take that as a sign that everything is working as expected.

Thanks!

 

[Johnny Fartpants]

Out of interest is this the system with 3 x 45bay JBODs? If so how many HBAs do you use and how are they cabled?

 

[fullspeed]

Out of interest is this the system with 3 x 45bay JBODs? If so how many HBAs do you use and how are they cabled?

Specs are in my sig, SuperMicro Chassis SC847J / LSI9206-16E HBA.

Hard to explain how it's cabelled without a bunch of diagramming but i'll give you a general breakdown. Each chassis needs to have the internal backplanes connected so all the disks can be seen then you sent it out through the external ports, I use the first external port for all connections coming in and the third external port for all connections going out.

I actually found my diagram I sent someone a year ago, This is the basics on how I got it working without any redundancy in the cabling. IN is external port1 OUT is external port3, the IN on the very first chassis is connected to my Dell Server which has my OS drive, SSDs and my HBA.

Hopefully this helps.

 

[Johnny Fartpants]

OK I think I've gotcha.

So you run one HBA with a single SAS2 cable connected from your server head to a 45 bay JBOD which has its internals connected together? Then off that JBOD you connect another 45 bay JBOD and off that one you connect another? All going via one single HBA and one SAS2 cable?

How many disks in each JBOD?

Did you ever have fast Scrub speeds? If so what speed?

Does your model of JBOD only have one input? Did you ever think about multipathing them to two or three HBAs for resilience and to improve performance so they're not all channeling through one HBA/SAS2 cable?

Also out of interest have you done any disk replacements on those JBODs, in particular at the back and if so did you find any issues?

Sounds like a very interesting setup. Will defo follow your threads with great interest.

 

[fullspeed]

OK I think I've gotcha.

So you run one HBA with a single SAS2 cable connected from your server head to a 45 bay JBOD which has its internals connected together? Then off that JBOD you connect another 45 bay JBOD and off that one you connect another? All going via one single HBA and one SAS2 cable?

How many disks in each JBOD?

Did you ever have fast Scrub speeds? If so what speed?

Does your model of JBOD only have one input? Did you ever think about multipathing them to two or three HBAs for resilience and to improve performance so they're not all channeling through one HBA/SAS2 cable?

Also out of interest have you done any disk replacements on those JBODs, in particular at the back and if so did you find any issues?

Sounds like a very interesting setup. Will defo follow your threads with great interest.

Yeah everything is a single SAS path, no double connections anywhere.. in and out. I haven't thought about multipathing them because it's not production critical and I'm bottlenecked by speeds in areas not involving my HBA/Backplanes. CIFS will only give me 50MB/s max per thread no matter what I do which is a long and painful story with no happy ending. If I have ever need speed I fallback to FTP and I've attained 650-700MB/s speeds that way.

I have done disk replacements and the only problem is there is no good way to find out which disk actually failed. Basically I replace the busted disk in Freenas then physically go down to my rack and look at my array, because my array is going 24/7 the access lights are constantly blinking. The dead disk is always the one that is completely dark, the alternative to this might be labelling every single disk with it's serial number (but aint nobody got time for that)

I've learned not to trust my scrub speeds, They go up and down.. from not moving at all to 4GB/s and back down again. For the scrubs I am using aggressive sysctl settings because of how long they take, I think if I grow my array any bigger they won't finish within a month :(

vfs.zfs.scrub_delay=0
vfs.zfs.top_maxinflight=128
vfs.zfs.resilver_min_time_ms=5000
vfs.zfs.resilver_delay=0

Feel free to ask more questions or look at earlier threads, it's been one year since I setup these arrays and I've learned quite a bit (mostly the hard way)

 

[Johnny Fartpants]

OK, thanks for the update.

50MB/s on CIFS sounds painful, whats that all about?

Regarding disk location then I may be able to help you there. Have you tried using the sas2ircu command? If not try this.

sas2ircu list
This will give you a list of available controllers on your system

sas2ircu 0 display
This will give information on all the drives of that HBA including their Enclosure number and Slot location (please note that enclosure numbers are different depending on which JBOD and if the disk is at the front or back)

sas2ircu 0 display | grep -B9 -A3 serialnumberofthedrive
This will give you information on the drive including its Enclosure number and Slot number

sas2ircu 0 locate 0:5 ON
This will turn the red light of the enclosure on so you can identify the location

sas2ircu 0 locate 0:5 OFF
Turns the light off

I hope you find this helpful.

 

[SirMaster]

For those mentioning FRAG, it doesn't have an effect on scrubs. It's not measuring your data.

FRAG is a measurement of the fragmentation of the free space. Scrub is scrubbing the data blocks. There is no measurement in ZFS for how fragmented your data is.

The FRAG property has to do with writes. If your freespace is too fragmented then writes will be slower.

If you want fast scrubs, use larger recordsizes.

On my simple 12x4TB RAIDZ2 with WD Red disks (which are some of the slowest disks) my pool scrubs my 26TiB of data in 6 hours at an effective rate of about 1.3GB/s. I'm using 1MiB records.

This is more than twice as fast as it was back when I was using 128KiB records.

 

[fullspeed]

OK, thanks for the update.

50MB/s on CIFS sounds painful, whats that all about?

Regarding disk location then I may be able to help you there. Have you tried using the sas2ircu command? If not try this.

sas2ircu list
This will give you a list of available controllers on your system

sas2ircu 0 display
This will give information on all the drives of that HBA including their Enclosure number and Slot location (please note that enclosure numbers are different depending on which JBOD and if the disk is at the front or back)

sas2ircu 0 display | grep -B9 -A3 serialnumberofthedrive
This will give you information on the drive including its Enclosure number and Slot number

sas2ircu 0 locate 0:5 ON
This will turn the red light of the enclosure on so you can identify the location

sas2ircu 0 locate 0:5 OFF
Turns the light off

I hope you find this helpful.

If you want a full rundown on my CIFS issue I have a full thread on it from last year, I've 100% given up. All four of my Freenas boxes are limited at 50MB/s per transfer, I've tried just about everything.

I ran sas2ircu list and it kernel panicked my box and rebooted it :/

 

[Johnny Fartpants]

Ouch that doesn't sound good. Sorry about that, never seen that happen before. I've ran that command on several FreeNAS systems and never seen that reaction. Saying that I've never had them cabled in the way yours is either so perhaps that's the variable.

Did you say you've done disk replacement from the back of one of the JBODs before?

How is the server connected to the network?

 

[Johnny Fartpants]

What version of FreeNAS are you running and what firmware is on your HBA?

 

[fullspeed]

Ouch that doesn't sound good. Sorry about that, never seen that happen before. I've ran that command on several FreeNAS systems and never seen that reaction. Saying that I've never had them cabled in the way yours is either so perhaps that's the variable.

Did you say you've done disk replacement from the back of one of the JBODs before?

How is the server connected to the network?

It's all good, it needed a reboot anyways. Yes I've done a bunch of disk replacements,(back and front) very easily. Regarding the network, both of them are on a 10GBe fiber using Intel X520 cards.

Freenas version is FreeNAS-9.3-STABLE-201512121950, HBA is using p16 firmware. I have another server with p20 and it doesn't solve any of my problems.

 

[Johnny Fartpants]

Regarding your 50MB/s issue do you get that speed even when adding data from a local machine to the Server so client A to server B style? I've seen people before connecting to two different servers on their desktop and try to move data from server A to server B and wonder why they're not seeing 1Gb/s speed.

 

[fullspeed]

Regarding your 50MB/s issue do you get that speed even when adding data from a local machine to the Server so client A to server B style? I've seen people before connecting to two different servers on their desktop and try to move data from server A to server B and wonder why they're not seeing 1Gb/s speed.

Right now I have maybe 12 database servers of various specs transferring to it directly daily and they are all on a 10 gigabit low latency network. They all get 50MB/s.

It's a lost cause and based on my conversation today I think we are going to scrap Freenas completely and use something else, I can't disagree.