Disk resilvering takes forever

[nicosalto]

Hi,
One of my hard drive in raid1 failed. So I replaced it with the same model (wd 4tb red model). The replication started 3 days ago and it's still not finished... That sounds like a long time to transfer around 3.5 TB of data from one hard drive to another on the same machine...
I run FreeNAS 9.3.
Anyone had similar issues before?

 

[wblock]

That is not replication, but resilvering. Is your 4TB drive really that full? Because overfull drives get a lot slower.

 

[rs225]

If you can run the command zpool status you should see how much time is estimated for completion of the resilver.

 

[nicosalto]

Thanks for correcting my title! I will try this command line tonight and see what's happening ;)

 

[nicosalto]

It really takes a long time, it is still going since last Saturday. according to the zpool command i am 76% done ...

 

[nojohnny101]

Please post a screenshot or picture of the zpool status screen. What is the rate of resilver?

 

[tvsjr]

You haven't answered the question above... how full is that pool? The output of zpool status and zpool list would help. Resilvering will suck if the pool is nearly full.

 

[RegularJoe]

is one a 512k sector drive and the new one is a 4k sector drive? Does the new drive have a firmware bug? have you done a diskinfo -citv /dev/xx to see if both perform the same? have you run a smart test short and long on the new drive?

 

[nicosalto]

Ok im not sure what is going on, it seem to resilver the disk in loop, restarting from scratch all the time.

as you can see it went back to 63% here....

this is my zpool list:
videos 3.62T 2.89T 749G - 30% 79% 1.00x DEGRADED /mnt


What do you think I should do? How do I stop the resilvering ?

regular Joe, to answer your questions:
is one a 512k sector drive and the new one is a 4k sector drive?
iam not sure, but both of my hard drive are the same model : WD40EFRX

Does the new drive have a firmware bug?
I don't know

have you done a diskinfo -citv /dev/xx to see if both perform the same?
no

have you run a smart test short and long on the new drive?
no

 

[tvsjr]

The drive that you're using as a replacement is throwing MASSIVE numbers of checksum errors. And, you have errors on your other drive as well... so, no matter what, you've lost data. The cause of the checksum errors could be a bad drive (this is why we encourage doing burn-in tests before placing a drive in service) or hardware problems - like a bad HBA/SATA controller, bad or undersized PSU, cable issue, etc.

 

[nicosalto]

The drive that you're using as a replacement is throwing MASSIVE numbers of checksum errors. And, you have errors on your other drive as well... so, no matter what, you've lost data. The cause of the checksum errors could be a bad drive (this is why we encourage doing burn-in tests before placing a drive in service) or hardware problems - like a bad HBA/SATA controller, bad or undersized PSU, cable issue, etc.

Ok I see... What should I do next? How do I stop the resilvering?

 

[nicosalto]

Should I send those hard drive back to wd?

 

[pschatz100]

I had a similar problem when replacing a disk, and the errors were caused by bad power to one of my drives. I was using a molex to sata power adapter for one of the drives and it was bad.

Make certain that all your connectors are good and properly connected - both sata and power.

 

[nicosalto]

Thanks for your answer Pschatz. I will try this too.
But i need to stop the resilvering safely first. It seem that it's not possible ... I've looked into the forum, but no one have a real answer for this.

 

[nicosalto]

My guess is i am gonna have to force stop it ?

 

[rogerh]

My guess is i am gonna have to force stop it ?

It might be best to select the new disk in the GUI disk view and offline it. If it won't do that because of the errors and running a scrub does not help then I agree it will be necessary to shutdown FreeNAS and physically remove it. Then a scrub might recover as much as possible from the remaining disk and allow a new one to be added.

 

[pschatz100]

In my situation, I never forced the resilver to stop. I did a system shut down and then checked all my cables and connections. I replaced the molex adapter with another one I happened to have on hand. When I powered up the system again, the resilvering restarted. I checked the disks under the volume status tab and the write errors were gone. Resilvering progress seemed to be better, so I let it run to completion. Then I ran a scrub on the volume.

At the end of the day, I don't know whether or not I actually lost any data. I haven't noticed any, but I do keep good backups - just in case.

 

[nicosalto]

Thanks!
yes so I did force shut down the server and then unplugged the new hard drive. Then when I restarted I wanted to scrub the volume. But I can't, FreeNAS still think it resilvering...

Code:

[root@freenas] ~# zpool scrub -s videos
cannot cancel scrubbing videos: currently resilvering

Now I am in the process of saving all my data so I can recreate a healthy volume from scratch...
what should i do with the new hard drive? should I return it?

 

[rogerh]

Test it in another computer (SMART long test, badblocks if you are feeling thorough). If bad, return. If good, look into cable or power problems. Or possibly MB or memory if you still have problems with the FreeNAS machine.

 

[nicosalto]

Thanks Rogerh,
I will test them with SMART long test. Since my data is now all offline, I was thinking to take the time to expand my pool with 2 extra 4T hard drive and switch to RAID 5 or 6. And also update to FreeNAS Corral as well.
I hope I wont have any more bad sectors issues!