Recovering Configuration After Hard Drive Swap

[William Bryan]

Hi all,

I know just enough about this to be dangerous!!!!

I had a system set up with four 3-TB hard drives in a RAIDz2-0 configuration. One of the drives was failing so I swapped it out with a 6-TB hard drive. I was trying to get the drive to resilver and mistakenly used the "Import Volume" option.

I now have three 3-TB hard drives in a RAIDz2-0 configuration and a fourth drive (the new one) listed as "stripe". My box will only accommodate four drives, so the failing drive has been physically removed. It still shows up under volume status in the GUI and in Shell as "Offline". My available storage space has increased from just of 5 TB to 9.9 TB, so I'm afraid the data on the three RAIDz drives has been spread out onto the new drive.

What is the best course of action to put my server back to the RAIDz configuration for all four drives?

Can I pull the new drive out and put the old one back in, then bring it back online? (It still functions, but was degraded.) Then take the new drive, format it, and put it back in? I don't know how I can remove the new drive. I've tried taking it offline, but it says there are no replicas.

All my data seems to still be accessible. The only sure way I can think of is to buy another 6 TB hard drive, put it in another box, copy all the data onto this drive, start the server over from ground zero, and then copy the data back on. This is definitely my least favorite option.

When you respond, please type slowly so you don't lose me. ;)

Bill

 

[joeschmuck]

You may be screwed my friend. Do not pull out that drive or your data might be gone. You have put yourself into possibly a very bad situation.
EDIT: New post coming, just wanted to get this to you fast.

 

[joeschmuck]

Post (in code brackets) the output of 'zpool status'. This will determine if you are screwed or not.

If your new drive is part of your pool, then things are not good. If it's a separate pool then things are good for you.

What to do if they are not good... Backup all your data now that you want to keep because you will be destroying your pool to fix your problem.

If things are good then you can destroy the single drive and then replace the failed drive.

Lets see what you have before moving forward though.

 

[William Bryan]

Here is what I have going on....

 

[William Bryan]

Here is the status through the GUI.....

 

[William Bryan]

Here is what I get when I click on "Storage". I had about 3.1 TB of data on the system before I opened the box. Could the new drive have simply copied some of the data off, but left it intact on the three original drives?

 

[joeschmuck]

So you are in the bad situation. Your data is being striped across the new 6TB drive and the [other 3 drives] as a group. If you lose the 6TB drive, you lose all your data.

The only recovery method you have is to copy all your data off your system to some other device (DVD-R, Another Computer, External Hard Drive connected to another computer, etc...)

I would backup the really important data first. If you have limited storage, well you will need to pick and choose what you can save. Once that is done you will need to destroy your pool and recreate it as a RAIDZ2 using all four drives installed.

Also, backup your configuration file, you shouldn't need it but you never know.

 

[William Bryan]

Thank you. I was afraid of this. I guess I need to figure out how to destroy my pool.

I ordered another 6TB drive and should have it by Tuesday. I can copy all the data onto it, destroy my pool, recreate it as a RAIDzs, and copy it back.

Looks like I have some reading to do on how to do all this. Do you know of any tutorials or articles explaining how to do this in simpleton language?

I just backed up my configuration too.

Thanks again for your help.

 

[joeschmuck]

The user manual actually tells you how to do all of this in chapter 8. It's not too much reading but it sounds like you have time. Also, backup any important data that you can now, you never know if the new 6TB drive you added will die of infant mortality.

EDIT: When you Detach the volume, ensure you mark the checkbox to Make as new (destroy volume). Look at figure 8.1L.

 

[William Bryan]

Thanks again for your help.

 

[William Bryan]

I appreciate your advise and have a few more questions (if it's okay).

So I purchased the 6 TB drive and I'm copying everything on the server onto it. (I also bought a PCI card with 4 internal SATA connections.) Once it's finished I will be destroying my pool. My son brought up an interesting option. Instead of running RAIDz2 (which I actually was) configure for RAIDz and then create two vdevs. Put two drives in each vdev. I would have to buy a 6th drive and once the server is set up and I've copied the data back onto it I could use the 5th and 6th drives and create a 3rd vdev. When I buy my 7th and 8th drives create a 4th vdev.


Vdevs Drives/Vdev Upgrading Results Increasing capacity

1 8 5 drives now, add each one as I buy it Most storage capacity, lower IOPs Capacity equals smallest drive of all 8

2 4 4 drives now, buy 3 more to make 2nd vdev Middle of road for capacity and IOPs To increase capacity swap all 4 drives in Vdev

4 2 4 drives now buy 1 more to make 3rd vdev Least storage capacity, highest IOPs To increase capacity swap both drives in Vdev


Can I add additional vdevs after I've added my data until I have a total of 4, or do I need to build all those now? His point was I would see an increase in data storage after replacing 2 drives instead of all of them. I would also see an immediate increase in capacity because vdev 3 would have 2-6 TB drives. vdev1 has a 3 and a 6 TB drive, vdev2 has 2-3 TB drives. If I lose 3 drives under a RAIDz2 I lose all my data (I can't see this happening very much as long as I monitor my system.)

From what I've read, IOPs are much higher with the multiple vdev configuration, but overall storage capacity is less. Speed isn't really that important to me, space is. Being able to upgrade little by little instead of all at once for more size is also a plus though.

Can you think of any downsides to any of these configurations that I'm missing? Can you add more vdevs like he thinks you can?

Thank you for taking a couple minutes from your day.

Bill

 

[William Bryan]

My table collapsed when I posted

 

[Bidule0hm]

Use the code tag to keep the multiple spaces and to have a fixed width font when you do some ASCII art ;)

 

[joeschmuck]

Can I ask you what you are using your NAS for? Also what do you feel the most intense application you are using on your server. I'd say this is how you should design your Vdev pools. I'm going to read your last posting again, I got a bit confused reading it the first time, but first I have dinner to grab from the Chinese joint down the street.

 

[William Bryan]

I use the server for several different purposes. I'll list them in order of importance.

1. I own my own small business and store finished information on my server. (I perform 3D laser scanning. www.eyeondetail.net)
   1. The folders for each project are rather large. Moderate sized projects are running between 40 GB and 155 GB. If things progress like I'm hoping I could very easily have project folders 10 times this big.
   2. Since it's for my business and includes not only projects for clients but tax information, I can't afford to lose the data.
   3. The data is mainly stored for archival purposes. I do all my processing using the hard drives on the machine. Upload and download speeds are not critical.
2. I keep pictures, videos, and music on the server.
   1. The bandwidth for these has been fine with the RAIDz2 configuration.
   2. It's just my wife and me, if we're watching a movie or listening to music, we are both doing it together.
3. I use the server to share data between different computers. Some of this is done for the business (work files for processing) and some for personal use files.

For these reasons I'm leaning toward one vdev with a the RAIDz2 configuration. I realize I will have to upgrade all my drives to see a change is storage capacity, but with 8 6-TB drives (according to your Capacity Calculator link) I will have just under 33 TB of data with single vdev instead of having 22 TB with 2 or 4 vdevs. The 50% increase in disk space is definitely a draw. It should take quite awhile to fill the space up and as drives fail I'll replace them with larger drives.

Also, if I lose both drives in a 4 vdev system, then I've lost data. If I lose any part of a job it could be the wrong part (if it's my raw scans I won't even be able to rebuild it from scratch). I realize the odds of having both disks go down in the same vdev is a probably a lot less than having 3 disks go down from a total of 8, but it also seems less likely I would have 3 drives fail all at once instead of 2.

Hope the Chinese is good.

Bill

 

[joeschmuck]

Since this is mainly a slow storage use requirement (streaming media is low IOPs) then I wouldn't do multiple vdevs. Running a RAIDZ2 or RAIDZ3 would be one of the options I'd choose.

As you are aware you should start with the number of hard drives in your vdev even if that means a mixture of hard drive sizes. So if you do not have eight 6TB drives, any mixture of 8 drives will get you started. If you can I would consider RAIDZ3 if you are going to really max out with large capacity hard drives due to the length of time it takes to resilver a large capacity drive. If you have a backup of your critical/important data elsewhere then it may not be as important to you however 33TB of storage would be crazy to restore if you had to. So it 33TB looks good, maybe 9 hard drives could work for you. Hey, it's only money. It's just when someone says business, I start thinking RAIDZ3. If you cannot add more hard drives to achieve RAIDZ3, well eight 6TB drives at RAIDZ3 would be 27TB (uncompressed) which is still a lot.

Just food for thought is all. And read around to make up your own mind how you want to move on this.

Chinese was good tonight, my belly is full.

 

[William Bryan]

Thank you for your thoughtful response. I wasn't even aware of RAIDz3. I'll read up on it. I'm pretty much stuck with 8 drives though, because my case will only hold 8 drives. I can't spend the money on a new case and everything else I would need to buy.

Right now I only have the 5 drives. At $300 a pop, I really didn't want to buy 3 more drives right now. I already bought 2 this week. My business is just starting out (January) and cash flow is still low.

Once I copy the data onto the this hard drive I will have to rebuild my system with just the 4 drives. After I copy the data off it, it will be free to put into the box. My son said he thinks I can add drives to the system as I go until I reach 8. Your statement makes me question this. What problems does adding additional drives present? I had planned on adding the 5th drive right away. If adding additional drives can be done, would it be better to wait until I have purchased 3 more and add all 4 in at once to make 8?

 

[joeschmuck]

You cannot add drives in the simple way your son thinks, not to a ZFS file system, but you can add more drives, you just need to do it the right way. Once you create your vdev it will be locked into 4 hard drives. If you create a RAIDZ2 then you can only increase the size of the drives to increase the size of your storage. The trick here is the vdev will only take on the size of the smallest drive so if you were to replace 3 drives with 10TB drives and still have a 6TB drive, your system will still look like all the drives are 6TB until you replace the last drive with a 10TB drive and then your storage size will jump up. This is because the data is striped (saved) on all the drives, not a single file goes to a specific drive. Hope I'm not confusing you too much. Cyberjock has a power point describing vdevs which isn't bad at all to read.

You know 10TB isn't a small amount of storage. If you need more then you will figure it out when the time comes.

And as you are very aware from your own recent experience, you added 1 drive to your vdev and put it entirely at risk. I'm glad you were smart enough to figure it out before you lost your data.

 

[William Bryan]

I guess I'm confused. I can't add drives in the simple way my son thinks, but I can add more drives if done in the right way, yet the vdev will be locked into 4 hard drives?

The storage demands for my business dictate more than 10 TB of data. (Can't believe I just said that!!!)

I considered buying the 3 more drives and sticking the failing drive in as a place holder, but that seems way too risky. The only other options are: buy 4 6-TB drive and use the one I'm building my server with as a spare; or buy 3 6-TB drives and a 750 GB "throw-away" drive as a space holder to build the system with. Once the system is built, pull the drive out and put the 6-TB drive in that I'm building the server from.

 

[Bidule0hm]

To explain differently, to have more space:

• You can replace all the drives by bigger drives
• You can add a new vdev to the pool (but you'll probably want to add the same type of vdev so for example if the first vdev(s) is(are) RAID-Z2 then you'll want to add a RAID-Z2 vdev so you'll need at least 4 more drives)
  
• You can destroy the current pool and recreate it with more drives
  
• You cannot add a single drive to a vdev to expand it
• However you can add a drive to a pool but you'll lose all the redundancy, as the drive will be effectively alone in a new vdev striped with the other vdev(s) of the pool, and it's probably not what you want

If the concept of vdev and pool are unclear then I recommend you to read cyberjock's guide, link is in my signature ;)