How to replace hard drives?

[MindBender]

One of the hard drives in my FreeNAS produced SMART errors last night:

Code:

(ada4:ahcich4:0:0:0): READ_FPDMA_QUEUED. ACB: 60 40 f0 06 f9 40 23 00 00 00 00 00
(ada4:ahcich4:0:0:0): CAM status: ATA Status Error
(ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 84 (ICRC ABRT )
(ada4:ahcich4:0:0:0): RES: 41 84 00 00 00 00 00 00 00 40 00
(ada4:ahcich4:0:0:0): Error 5, Retries exhausted

From what I've read, this is pretty serious. Because of this, and because my (5x 3TB Seagate Constellation ES.2 Enterprise grade) drives are over 6 years old now, the time has come to replace them. I am thinking about buying 5x 6TB WD Red Pro, but nothings has been decided yet.

So now I need to backup my main pool to my set of new drives, but unfortunately I can only connect one set of drives at a time. So far I have connected an external 8TiB Seagate Archive drive, created a pool on it, and now I'm copying everything from my main pool to my archive pool.

But while this copy is running, I'm hoping there must be a better way, and that you guys can point me into the right direction. Because my method is very cumbersome; My main pool was my only pool until now, so besides my shares, it also contains jails and who knows what else. Verifying this backup after copying is a lot of work, and restoring everything back to a new pool probably even more. I could do this a second time, on a second disk, to easy my nerves on having everything on a single cheap disk for a while, but still it won't feel right when detaching my main pool on the old disks.

Is there an easy way to make a checksummed backup of my main pool on an external disk (twice), verify these backups, replace the main pool drives, restore the backup and verify it?

 

[Chris Moore]

I am thinking about buying 5x 6TB WD Red Pro, but nothings has been decided yet.

Bang for the buck, the 8TB HGST drives are better. I have been doing some comparison shopping for a purchase at work. This is the best price I found:
https://eshop.macsales.com/item/HGST/0F29801/

while this copy is running, I'm hoping there must be a better way, and that you guys can point me into the right direction.

You can simply replace one drive at a time and once all of the new drives are resilvered into the pool, the pool will autoexpand. I have done that at least three times over the years. Works fine.

 

[Chris Moore]

Alternatively, you might want to consider these WD Gold drives:
https://www.allhdd.com/hard-drive/s...KCEYQlyP3uf4PxcahKkGdP21meWAsqqhoC-awQAvD_BwE

 

[MindBender]

Bang for the buck, the 8TB HGST drives are better. I have been doing some comparison shopping for a purchase at work. This is the best price I found:
https://eshop.macsales.com/item/HGST/0F29801/

Unfortunately they're unavailable here in the EU. And I'd rather stick with enterprise grade drives, because I don't have an interface left for a 6th (2nd redundant) drive.

You can simply replace one drive at a time and once all of the new drives are resilvered into the pool, the pool will autoexpand. I have done that at least three times over the years. Works fine.

Interesting: I used to do that with my Thecus and ReadyNAS drives. But isn't that risky?
And will the pool and dataset expand after all drives have been replaced with bigger drives?
I suppose it's best to start with the drive producing the error: Is there a way to identify it? ReadyNAS offers a feature to make a drive flash its LED.

 

[Heracles]

Hi MindBender,

I confirm to you that Chris is right : once all your drives are replaced with bigger ones, FreeNAS will auto-expand your pool to the new and bigger size. Just like Chris, I also did it a few times and it always worked perfectly.

The moment your pool has some redundancy, you are free to go. In the WebUI, you mark one drive as Offline, remove it, replace it and back in the UI, mark the new drive as the replacement of the previous one. If your system supports hot plugging drives, you can do it live. If not, you need to power down before removing / plugging drives. Of course, because you have one drive that is failling, start by replacing that one.

Should your pool have no redundancy, you can not remove a drive like that. But honestly, a pool without any redundancy is not worthy of hosting anything but test and temporary data you are ready to loose at any time.

Also, if your pool has only 1 redundant drive (RaidZ1), know that you will be vulnerable during the re-silvering process. Your redundancy will drop to Zero during that moment. This is one of the reason to go at least RaidZ2 for your pool.

Good luck in your upgrade process,

 

[MindBender]

Thanks guys! Offlining the drive is a good suggestion. It will also switch off the LEDs, making it easy to identify the drive. Fortunately I have a redundant drive, and fortunately my system supports hot-plugging.

My USB drive copy failed, unfortunately; The drive seemed to have disappeared from the USB bus and the shell copy command has been stuck the whole night. So it seemed like a good idea to shutdown my NAS until the replacement drives are in. But I'm afraid the shutdown is stuck on 'something'. It may the copy, it may be the USB drive - which I have unplugged to make sure it's gone - or it may be still flushing the ridiculous amount of RAM (128GiB) I have installed. I will leave it for today, and check again tonight when I'm back from work.

Thanks for the help guys!

 

[Chris Moore]

Interesting: I used to do that with my Thecus and ReadyNAS drives. But isn't that risky?

The level of risk depends on the level of RAIDz you are running, to some degree, and the health of the existing drives. Also, if you have an extra bay, you can do a 'live' replace of a drive that is currently in the system. Meaning that you can add an extra drive to the system, select the defective drive in the GUI, tell FreeNAS to replace it with the 'extra' drive and wait for the resilver. It takes longer for the resilver to complete that way, but it does not reduce redundancy. Once the resilver is complete, the drive that has been replaced is offlined automatically.

Is there a way to identify it? ReadyNAS offers a feature to make a drive flash its LED.

The most reliable way to identify your drives is by serial number, but if you have an enclosure with good activity LEDs, when you offline the drive, the activity should stop. That might help you to ensure you are working on the correct drive.

My USB drive copy failed

It has been my experience that USB has always been unreliable in FreeNAS. If you really want a backup, it would be better to connect the drive by SATA.

What is the pool configuration? RAIDz1 or RAIDz2 or something else?

 

[Chris Moore]

Unfortunately they're unavailable here in the EU. And I'd rather stick with enterprise grade drives, because I don't have an interface left for a 6th (2nd redundant) drive.

Sorry, I didn't notice this part of the post before. The HGST 8TB drives are enterprise grade drives, the ones I linked to are being resold after removal from a data-center for replacement with larger drives. I am looking at those for use in my home NAS. For work, we are looking at 12TB drives.
What kind of chassis are you working with? I am guessing SATA ports? If you are out of interfaces, you can always add a SAS controller. I run all my storage drives from a SAS HBA card and find that it works very reliably.