Why is my scrub taking so long?

[ethereal]

I have a weekly scrub of one of my pools running on sunday and takes approximately 21h.

Today the scrub is only at 19.12% - 2.44T scanned out of 12.7T at 53.3M/s - 56h20m to go.

I don't have anything unusual running no heavy reading or writing to my pool. i'm using the same hdds and there are no problems (running long and short smart tests but not during a scrub).

any ideas? things i should investigate?

thank you

 

[toadman]

I've had a bad disk drag the pool performance to a crawl. You can investigate the i/o rates to each disk doing some dd activity to the pool. jgreco has a good post here somewhere about checking for bad disks and or disk speeds. I located a poor performing disk that way. Replaced it and the pool was back up to expected speed.

How fast do you think the pool should scrub?

 

[ethereal]

all the previous scrubs took 21 hours - i checked the reporting tab and all 6 disks were 6 Mbytes/s. At 3:45 pm they started to speed up and now they are at 70 Mbytes/s. Still don't know why. All the drives were at the same speed - so i doubt it is a bad drive - they have daily short tests and weekly long tests and there have been no problems reported.

 

[ser_rhaegar]

I've had the short/long tests hit when a scrub was going on and everything went to a crawl. You might check to see if smart tests were running during the scrub. I schedule in a way to make sure they don't run at the same time anymore.

 

[ethereal]

i should have said none of the smart tests are running while i am scrubbing

 

[toadman]

Pool fragmentation and a lot of small blocks can slow a scrub, as can any i/o access to the pool. Has the data changed much since the last scrub?

 

[Robert Trevellyan]

All the drives were at the same speed - so i doubt it is a bad drive

A vdev will perform at the speed of the slowest component drive, so you will always see the performance of all drives be virtually the same during a scrub. You would have to test the drives individually to isolate a bad one.

 

[joeschmuck]

How full is your pool? (including snapshots if you use them)

 

[Mirfster]

Get @DrKK to give you an estimate... He has an uncanny ability to provide very good estimates on scrubs... ;)

 

[joeschmuck]

Get @DrKK to give you an estimate... He has an uncanny ability to provide very good estimates on scrubs... ;)

He already said it normally takes him 21 hours to do a scrub. That sounds like a lot of data to me. But you are correct, @DrKK does have that ability.

 

[ethereal]

8.5 Tib - 80% full - no snapshots

the last scrub took 35h

 

[joeschmuck]

Probably a lot of fragmentation I suspect. 80% full is quite a bit.

While it appears the system is getting slower at running a scrub, can you correlate it to that you may have been adding significantly more data over that period of time (less free space)?

 

[ethereal]

it has been 80% for months and for as long as i remember the scrub has taken 21 hours.

the last scrub seemed to speed up around 3:45pm and got progressively faster

 

[joeschmuck]

Very odd. Hopefully it's not a bad drive but I'm sure you will check into that. Maybe it's a failing SATA cable which is much cheaper to replace.

 

[joeschmuck]

Hey, what drive model specifically is in your pool that is being scrubbed? Please don't tell me that you are using AV drives in a NAS system. AV drives operate much differently than a normal hard drive meaning they don't care about errors the same way that a normal hard drive does. AV drives will try to get your data but after a few tries it gives up and moves on to the next piece of data. Well that is for my WD Purple drive which is why I use it in my DVR. If some data isn't there in time then I want it to move on because video can handle a little loss without any significant impact to your viewing enjoyment. In a computer system that isn't the case, you want the drive to try hard to get the data, even if it takes a long time.

Let me know.

 

[DrKK]

A rule of thumb, for well-performing, good condition drives, is about 1-2 hours per TB of *USED* net filespace. There are a lot of variables, but just a quick glance at some of the FreeNAS I manage around the world---here's a small one, it's got 650GB used only, it takes 45 minutes to scrub. Here's another with 1.7TB used, scrub time was 2h55m. Right inline with what I was saying. These are, of course, fascistly maintained, drives in virginal condition, pools, monitored vigorously by DrKK. Here's another with about 4TB of data on it, scrub time right about 6.5 hours. Some caveats:

• These are all RAID-Z and RAID-Z2 configurations, it's possible a "mirror" would be faster?
• Most files on all of these are larger media files, not millions of small files.
• All of these systems have WD Reds (I use them exclusively---if I build you a system, you get WD reds, or you get someone else to build it)
• All of these systems use G3220-family CPU's on Supermicro X10 platforms
• Resilvers, and scrubs, tend to ACCELERATE (for some reason) over time. Thus, the estimates for "time left" often are unreasonably high at the beginning.

 

[Robert Trevellyan]

AV drives will try to get your data but after a few tries it gives up and moves on to the next piece of data.

I keep seeing this stated, but from what I've read, that's only true when the host issues the special ATA streaming commands that allow the drive to do that.

Example: http://www.wdc.com/wdproducts/library/other/2579-772012.pdf (search for "streaming")

 

[joeschmuck]

I keep seeing this stated, but from what I've read, that's only true when the host issues the special ATA streaming commands that allow the drive to do that.

And I thought it was up to the drive electronics to determine that. That was an interesting read and puts to rest a few issues that were in my mind, but still, I'd never use a AV drive in anything but an AV product. It's interesting that it states to have several copies of the data on the drive in different physical locations and to check it daily to ensure it's not corrupt and then to use the proper ATA commands to access it vice ATA streaming commands.

 

[ethereal]

i continue to have this problem. i have replaced the mini sas cables, the hba and the power cables to the drives. i think the next thing i should do is test the hdd in the Storage pool. this pool is RaidZ2 (6 drives). What is the best way to find a bad drive or eliminate them from the enquiry ?

 

[joeschmuck]

The link in my signature titled "Hard Drive Troubleshooting Guide" will help you figure out if one of your drives are failing.