Raid z server-rebuild resulting in 1 HDD loss, How do I replace new drive

[Donny Boshoff]

Good evening guys/girls

I'm new to FreeNas problem solving. I've been running FreeNas servers for the last 3 years upgrading regularly, with no hardware failure to my luck.
However, today, after replacing my hardware with a new HP Micro-server, I tried to import the array like always, now realizing that it only picked
up 3 of my 4 Hdd used in the Raid z array. Now, if this was a personal server, I'd just restart collecting data, but this is a company backup server
and I'll probably have some scary problems if I can't find a solution.



Well, all the info I can think of.

1.) I wiped the previous FreeNas setup from the Usb-Drive, installed FreeNas 8.2.0, also replacing it with 8.3.0 Beta to see if there's something else
I could use to get the data back.
2.) When trying to Auto Import Volume, It picks up the array / Disk but gives me the following error: The Volume " " failed to import, for further details check
pool status..... This is with both the remaining 3 hard-drives as well as with the 4th new HDD added.

Please help

 

[joshg678]

Well i am still new to freenas zfs etc.
But it would be failing because it only sees 3 drives.
Have you done the basic trouble shooting to find out why the 4th drive is not showing?

Is it showing in the BIOS?
You make sure all 4 drives have been connected with power and the data? (simple i know, but everyone has their moments)
If all is connected, i would (while off) disconnect and reconnect everything and see if anything changes

If not, i would switch some cables around and see if that makes a difference. Try replacing them with new cables


Its also very possible that the 4th drive died when it was powered off the last time. Or the cable connection went bad.


I think you use the -f option to force the import even when all the drives are not available, but i wouldn't do that, i'de try and get FreeNAS to see all 4 drives first.

 

[paleoN]

However, today, after replacing my hardware with a new HP Micro-server, I tried to import the array like always, now realizing that it only picked
up 3 of my 4 Hdd used in the Raid z array.

What happened to the 4[sup]th[/sup] HDD? You start talking about a 4[sup]th[/sup] new drive. Did the old one fail, did you decide to ram a new drive in there for the hell of it or ...?

From a SSH session as root paste the output of:

Code:

zpool status -v

camcontrol devlist

gpart show

glabel status

Throw the output inside of some [code][/code] tags as it will preserve the formatting and keep my eyes from crossing.

 

[Donny Boshoff]

Hi Guys

Thank you for repelling. To start at the top.

Hi joshg678,
Yup, I did troubleshoot the hardware first, I think the drive broke while shutting down the server.
I've tested the cables and Sata ports on the board by switching them out, one drive is broken, doesn't even pick up in the bios.
Before forcing anything I thought I'd rather talk to someone who knows more that me about fixing the problem.

Hi paleoN
I did see the 4th hard-drive was dead, so I tried to import the array of 3 remaining Drives to the new FreeNas, only getting the error
posted. I know of a "replace faulty Drive" function available, when a drive dies and your array is still loaded on FreeNas. I inserted a
new 4th drive hoping it would import the array with the new drive ready to rebuild. I hope this answered your question.

Please let me know if I need to post any specific information needed.

 

[joshg678]

That makes everything more clear then.
Assuming, you currently have the 3 original drives, plus the 1 New replacement drive attached.

Importing the Pool via the GUI will Fail, because it is unable to find the 4th drive, and currently (as far as i know) the GUI is unable to handle this.




So basically you will have to import the pool from command, and then replace the old bad drive, wait for re-silver to finish, remove the old drive, export pool, then finally import from the GUI.

Its something like this:

Code:

zpool import -f POOLNAME     (or -F cant remember which one)

then

zpool status POOLNAME

at this point you should see your pool degraded with the bad drive missing

you will want to make sure you get the device ID for the new drive, which you should be able to do from the GUI

Then replace the drive

Code:

zpool POOLNAME replace 0a0a0a0a0s0s ada3

wait, you can monitor the status of the replace

Code:

zpool status POOLNAME

once its finished

Code:

zpool detach POOLNAME olddevicename  (ex 0a0a0afdjdj292929)

then check the status again, make sure its says healthy
I would then srub it one more time to double check all is good

Code:

zpool scrub POOLNAME

then 

zpool status POOLNAME

wait till its complete

Code:

zpool export POOLNAME

then try to import from the GUI and see what happens.





I would like to double check with some one else to make sure this is correct, i can double check this procedure on my test VM to make sure this works

 

[joshg678]

OK
I just tested the above steps and it worked for me like a charm

However,
after words i realized that the new drive didn't have a Swap partition on it.

I would make sure you have your data backed up before continuing. All my 4gb of test data was in tact, but i would double check your backup first.
So to fix this i did this, since it was already in the GUI

I replaced the disk with its self.
Click the replace button, choose Inplace (the new disk)

it will tell you an error occured

I should show as offline
then replace it again

This time it should show, replace None with da6 for example

From command line, make sure the resilver finishes

then detach the offline drive from the GUI

Everything should be good now.


Scrub it for good measure.

 

[paleoN]

Please let me know if I need to post any specific information needed.

Reread my post.

Importing the Pool via the GUI will Fail, because it is unable to find the 4th drive, and currently (as far as i know) the GUI is unable to handle this.

This has not been my experience. A degraded pool should import.

Code:

zpool import -f POOLNAME     (or -F cant remember which one)

then

zpool status POOLNAME

This is ignorant. Don't import a pool with -f without trying a normal import first.

I realize you are trying to be helpful, but you have been wrong and the advice sub-par. You likely can replace the disk via the GUI anyway and it will create the partitions for you. Be careful with your suggestions on other peoples data.

 

[Donny Boshoff]

Hi Guys

paleoN
No, I didn't just ram the new drive, I did however shutdown the server and insert the new drive seeing that the old one didn't pick
up in the bios and I didn't have the previous setup left on the USB stick, after re-installing the new FreeNAS from scratch. I ran the tests
today after removing the new drive again, screen shots follow.
View attachment 1279 View attachment 1280 View attachment 1281 View attachment 1282

When using the GUI interface, clicking "Storage; Auto import volume; I find the option to select Disk: Datastore [zfs, is=17076....] but
then, clicking on import volume I receive the following error: The volume "Datastore" failed to import, for further details check pool status.
I'm thinking this means it's struggling to import the volume with the missing/faulty drive.

Please excuse me, I'm not big on FreeBSD shell yet, so I didn't really do much except for worrying.
I can try the zpool import -f POOLNAME command but I wont have space to backup the drives before I do.
If this is a big risk, I'll wait for other suggestions before doing so.

joshg678 , thanks for making sure I know this needs to be backup up before using your solution.

 

[paleoN]

No, I didn't just ram the new drive, I did however shutdown the server and insert the new drive seeing that the old one didn't pick
up in the bios and I didn't have the previous setup left on the USB stick, after re-installing the new FreeNAS from scratch.

You already explained this.

I can't help you further without the output of the commands I requested. Why? Currently I have no idea what state your zpool is in and neither do you.

 

[Donny Boshoff]

Lol, dont know why it didn't take, sorry

 

[paleoN]

You only have 3 disks showing up. The 4[sup]th[/sup] is still missing. What's more I would guess ada1 is the new blank disk as it has no GPT partitions.

SMART info for ada1:

Code:

smartctl -q noserial -a /dev/ada1

Output of:

Code:

zpool import

This will list the available pools. It won't import them and you should not force the import at this point.

 

[joshg678]

This is ignorant. Don't import a pool with -f without trying a normal import first.

I realize you are trying to be helpful, but you have been wrong and the advice sub-par. You likely can replace the disk via the GUI anyway and it will create the partitions for you. Be careful with your suggestions on other peoples data.

Thank you for your rugged criticism. May i ask why is it ignorant to force the import?

I see he only has 2 of the 4 original drives for his RAIDZ, so wouldn't the forced import fail anyway, or would it run the risk of corrupting the data?

My Apologies if my suggestions were poor, I am more accustom to working with data that is located in 3 or more paces.

 

[Donny Boshoff]

Ah guys

Thanks for still replying, I'm kinda getting worried, lol.
paleoN, The original drives where still in there, so I hope I missed a drive and setup the Raid-z over 3 instead of 4 drives.
I've replaced the dead drive, started up the server and redid the tests. Pictures follows:



I'l even pay someone to log into the server and repair the problem, that's if it's repairable.

Hope to hear from you soon.

 

[joshg678]

It looks like when you created it, you had 4 disks.
But for whatever the reason, only two of the disks you have connected show data, the other two are showing as blank.

Do you have all the orignal disks?

 

[Donny Boshoff]

Hi joshg678

Yes, I do have the original disks.
Just a recap by steps from the beginning.

1.) FreeNAS 7 installed on Usb-Drive; Gigabyte Motherboard with 4 x 500Gb Drives on Riad-z
2.) New HP Micro Server
3.) Wiped Usb-Drive and installed FreeNAS 8.3.0
4.) Installed the 4 x 500Gb Drives; started up to Auto Import
5.) Found one drive did not pick up in Bios; tested, drive completely dead
6.) Can not import Drive Pool because the drive is dead; so no replace drive function.
7.) Removed dead drive from Pc, replacing with New 500Gb Drive

Guys, this is really freaking me out... Maybe I setup the raid over 3 Drives instead of 4, and
now one broke. I should still be able to import the Pool and repair the data array, right?
I've checked all over, This drive was not swapped by accident and the server was working
perfectly right up until I replaced the hardware. Damn Seagate....

Kind Regards

 

[paleoN]

Why did you move the drives to the HP Micro? What's the story with ada1? Is it a new drive or not. If not then setup SSH, like I already suggested, and show the output of the smartctl command I gave you.

Guys, this is really freaking me out... Maybe I setup the raid over 3 Drives instead of 4, and
now one broke.

Output of zpool import shows the pool is made up of 4 drives.

You could always move the drives back to the old server and try to access the pool there.