Replacing all 4 drives under ZFS - Guidance

[feleven]

My FreeNAS 9.2-RELEASE-p3 (I assume this is 9.2.1 ?) has been running on a box with 4 x 1TB Hitachi Deskstar 7K1000 drives, vintage 2008 or so. I knew they probably didn't have a lot of life left, but they seemed ok to get my feet wet with a FreeBSD file and media server. Worked pretty well since November. I recently added a new 500GB drive to act as a backup, which I xfer data to using rsync. My ZFS volume/pool (still not sure which is correct) server is /mnt/FN-1TBMir, and I rsync everything below that directory to the backup drive. Not sure if its necessary, but better safe than sorry.

This morning I find an email stating that my drive array is running degraded. Referencing several posters in these forums I checked the Volumes and Disks, and ran the usually recommended shell commands:

zpool status -v
camcontrol devlist
gpart show
smartctl -q noserial -a /dev/adax (once for each drive in my system, from ada0 to ada4)

Sure enough, one of the old Deskstars had taken a dive. Reading more posts here concerning hardware selection, I chose to replace that drive with a WD RED 1TB WD10EFRX. As I stood in line at the store, I realized this was my opportunity to replace all 4 Hitachi's with current technology true-NAS-purposed drives to avoid another drive death in the near future, so I bought 4 1TB REDs. "In for a penny, in for a pound" as they say.

Before I start in on the replacement process, and being a noob with more courage than knowledge, could anyone here give me the Coles Notes version of the process for swapping out all the old drives and installing the new? As I said, I have a full backup from last night available, and the remaining 3 Deskstar drives are still functional, so other than replacing the dead drive first (I assume), what is the process for replacing each of the other drives? Just do the same thing as for the first drive x 3? In what order - dead drive, dead drive's mirror drive, then the other two mirrored (still good Hitachi's) in any order?

I know damn well that I dodged a bullet here, as prior to last week I hadn't done a full backup - I got lucky. So I'm a little twitchy about doing this wholesale drive swap-out - I don't want to squander a bit of good luck with bad procedure now.

Any suggestions? If you need more info, just ask and I'll post it. I did a quick search of posts about replacing drives, but all I found were about replacing 1 bad drive, or replacing drives without a backup. I'd feel more secure if someone could guide me in doing this right with the fewest assumptions possible. I have a knack for stumbling into the only pothole on a smooth road.

Thanks for any help. These forums have saved my butt more than once already!
Rod

 

[joelmusicman]

Frankly, if you're going thru the hassle of replacing all the drives, why not a capacity upgrade as well? For the drives alone, 3TB is the sweet spot for TB/$. If buying a new system, 4tb drives still deliver the best delivered TB/$ when looking at whole system cost.
Using Amazon prices (not including tax):
4x1tb: $280 / 2tb = $140 per delivered TB.
4x2tb: $400 / 4tb = $100 per delivered TB.
4x3tb: $520 / 6tb = $86 per delivered TB.
4x4tb: $752 / 8tb = $94 per delivered TB.

Also, I'm a bit confused about your config. It looks like you're running two mirrored pairs of drives, are they in the same zpool? If so, I think most of the experts consider this a "non-optimal" config, because even though you have two parity drives, if the wrong two go out, your pool is still sunk. Even right now you have zero parity protection for the pair with the bad drive. RAIDZ2 is much superior for data protection.

Of course, switching would mean nuking your existing pool.

 

[feleven]

Thanks for responding, Joel - greatly appreciated!!

Well, being a noob, when I added the 2nd pair of 1TB Hitachi's, the FreeNAS GUI applauded Extending a Volume and hailed the automatically chosen setup as being the optimal configuration. So I said fine. If the FreeNAS GUI isn't offering the optimal solution even when it says it is, how am I supposed to know that - if its NOT optimal, then WTF? Reading through these forums as a noob, before adding the drives, was of little help: everybody has a different configuration of different drives with different priorities and a different opinion on what's great and what sucks. I did read praise for RAIDZ2 on the forums, so assumed that if the GUI were going to promote what's optimal, that a RAIDZ2 configuration would be recommended if it suited my system. If it didn't, who am I to dispute it? I'm a noob, remember?

Anyway, let it go, I understand and appreciate your comment. I'll look into the RAIDZ2 option as you suggest, and if I like it, I'll nuke the pool and start over. This is why I did a test setup with the old drives to begin with - to make mistakes that don't matter in the long run. Apparently my drive configuration was a mistake, and I now know I can't trust the FreeNAS GUI's recommendations 100%. Thanks for the heads up! :)

I looked at drives ranging from 1TB to 3TB, and the prices on Amazon.ca for the WD RED drives were: 1TB:$85, 2TB: $125, 3TB: $160. The price difference between locally available drive sizes was even more. I wanted to get the bad drive replaced as soon as possible (based on dire warnings posted in these forums, and realizing as you say that if I lose the 2nd drive in the mirror, I'm screwed), so waiting 10 days for Amazon to deliver a replacement wasn't high on my wish list. So I bought local.

When I started this project, the objective wasn't to build the most powerful private server in the neighbourhood, and being retired my budget for what is essentially just a toy is limited. I just wanted sufficient NAS to provide file and media services to my home network used only by myself, plus a little space for backup of a couple other machines with modest needs. The drives I had were 1TB, and the files I'd already loaded onto them only consumed ~90GB. I don't store any ripped DVDs or BDs, as I prefer to watch my movies from the original discs, or from online services, and I no longer buy new movies on disc. So I really don't need an enormous expanse of storage - even with all my ripped music and all my full sized photos uploaded, I'm only using about 7% of the 1.7TB available. It made no sense (to me) to power and maintain all that extra drive space for the next 2-3 years, which in all likelihood I'll never come close to consuming. Again, I appreciate your trying to optimize my finances, but I made a conscious decision to buy what I did, based on the budget reality I live in, and my anticipated needs. I'm an engineer, I know how to assess cost vs features vs benefits against a list of prioritized requirements.

So - if I'm going to nuke the whole pool, then I guess my original question is moot - I just load the new drives into the system, ignore the FreeNAS GUI's commentary about what's optimal and go with RAIDZ2, and then - what? perhaps rsync from the backup? If I nuke the existing pool, and go to RAIDZ2, does the backup care how the pool is configured? can I just rsync the backup files onto the new pool?

Thanks again for your recommendations, Joel. They're a good start. Any other comments are still welcome.

 

[joelmusicman]

With only 7% (50-100gb I'm guessing?) used, nuking the pool is probably the better option. Test your backups before doing that though!

Re: Finances - you're probably not going to run out of space anytime soon so "belay my last" on that comment. :)

About the drives being non-optimal: 2x2 mirrored pairs is definitely not the worst choice, and from a phased approach probably made some sense. Since you can relatively painlessly start over, RAIDZ2 is just much better from a data protection point of view.

 

[feleven]

I'm doing my reading on RAIDZ2 as I write... :) ... I take it from your first comment that I can indeed use rsync to load the existing backup onto the new pool. That makes the process of replacing the drives relatively trivial.

Again, my thanks for your advice. Personal experience from others adds confidence to what I read here, and in the manual.

 

[joelmusicman]

You *should* be able to. That said, If it were my data I'd also make another backup by simply pulling the files to a different PC.

 

[feleven]

Understood. I can implement that extra safety step as I read and plan. Thanks! :)

 

[cyberjock]

I'll just pipe in that joelmusicman is spot on. So now you have a little more confidence in what he's writing. :P

 

[feleven]

It wasn't so much that I was lacking confidence in his advice, as in what I thought I understood from reading the manual and the postings in these forums. I never doubted Joel's recommendations were based on facts and observations, instead of assumptions. But manuals are hard to write so they're useful for new users (I've written my share for software and it took almost as long to write them as to write the software they described), and even harder to read and understand as new users. Having somebody say, "Well, this is what that section REALLY meant to say" is a big help, either in preventing a mis-step or in trusting that you DID understand what you read.

I'm a little nervous about this "nuking" of the existing pool - in this instance, does that mean just shut down the box, yard out the old drives and install the new? I obviously want to leave the backup drive in place physically and untouched, while having the new drives be recognized at boot up as available for initialization for new storage. So the sequence in my mind should be:

- save configuration
- shutdown
- yard out old drives
- install new drives
- boot up
- configure the 4 REDs into a RAIDZ2
- then, hmm...

Is it at that point that I restore the Saved Configuration, before rsync-ing the backup drive onto the new pool? I guess this question comes from not knowing exactly WHAT is stored in the configuration .db file.

 

[feleven]

Well, that was a bit hair-raising, even though the world wouldn't have ended if things had gone into the ditch. I'm back online with the new drives (GEEzus - between the larger 64MB buffers and SATA3 speeds of 2 of the 4 new drives, these things scream!! :) ), and although there were a few stumbles, I believe its set up correctly. For anyone following my breadcrumbs...

1. Saved configuration
- note1 : to my amazement, the configuration was NOT erased when I removed the server pool and rebooted - I thought it would be - so this is a pure safety move
2. shutdown the server, pulled out all cables, carried to workbench and opened up both sides to get at drives
3. removed 4 old drives, marking the one that caused the problem as NFG
4. installed 4 new drives, after marking S/N's on ends so I can identify without removal later
5. connected 2 new drives to SATA 3 PCIe card :D )
6. connected 2 new drives to motherboard SATA2 connectors )
7. left backup drive connected to motherboard SATA2 connector just as it had always been
8. held my breath and re-booted
9. Used ZFS VolMgr to configure the 4 drives as RAIDZ2 by sliding that little dial icon UP as well as to the right (changing from 2x2 to 1x4)
- NOW I understand why I didn't see a RAIDZ2 option on the drop-down box before, with only 2 drives on the initial install - with 4, its available
9a. changed the name of the pool from original, which I knew was going to cause grief later with shares, jails, plugins, etc
10. the old mirror/striped pool was still there, so I just Detached it
- wasn't quite sure what this would do, but other than complaining about shares still connected, the old pool was removed from the table amidst the red glare of the screen
11. configured a cron job to start in 2 minutes to rsync the backup data onto the new pool
- just reversed src and destination paths (new destination path - new pool just created with new name)
12. after backup was restored (checked from shell doing spot checks down directory tree), Uploaded Configuration.
13. rebooted
14. WTF happened to my new pool volume and restored dataset?? [lots of cursing and scrambling around]
15. Ah ha - I'm an idiot! the configuration file I uploaded had the old pool name and path embedded everywhere, and knew nothing about the new pool volume but expected to find the old pool volume
16. repeated step 10
17. Auto-Imported the new volume - the GUI found it, offered it in the drop-box, and it came back intact. Whew!
18. Went through all FreeNAS configuration pages (don't forget jails and services) and made sure all pool names and paths to it were changed to the new one
19. Saved configuration !!
20 rebooted
21. re-started Plex plugin (this will fail, obviously, if the plugin configuration still points to the old pool name path for media)
22. rebooted again for good measure

Others wiser than I will have to confirm if I did this correctly (except for the Upload Config snafu), or point out errors - ie. I probably should have detached the old pool before saving the configuration and shutting down. But whether by luck or good guidance, my system is up and running, and I can play my media files across the house network. The CIFS shares still act weirdly from my MAC, but no weirder than before. And I can use the Mac's Finder/Go/Connect To Server... to connect to both the server and backup pools without issue. Data transfer across the network is much faster with the newer tech drives. Success ! (well, I hope - ask me in a day or two).

Rod
[edit: thinking further about the SATA3 card and drives - I suspect that the speed increase I think I see is due to the bigger drive buffers (64MB new vs 32MB old). I don't think my home gigabit network could support the full bandwidth of a SATA3 6Gb/s data flow. Oh well, it sure feels damned fast now! :) - Rod ]

 

[ECCfrenaslover]

Sounds like you got it!

 

[feleven]

Sounds like you got it!

:D