Restoring from Replication

[Tony T]

I have two Dell 2950 servers with Quad Core Xeon processors, 16 Gig RAM and 4 2TB SAS drives runnig on a PERC6IR with all the RAID feature turned off as per the FreeNAS documentation. Each server boots from a 16Gig USB drive. I'm running FreeNAS 9.3.

One of the servers is my active NAS and the other is a backup NAS for the first server. I have a Snapshot and Replication task that runs every night and seems to work perfectly.

My question is this.... In the event of a major disaster where a complete machine fails (not very likely but possible), how do you restore the Active machine after the repair or replacement of the machine? This seems like a simple question but I've spent three nine hour days searching this site and Google for an answer. It may be that I'm a simpleton and the answer is so obvious that it doesn't warrant a reply but I need the peace of mind that I can recover it need be.

I have a backup for the USB drive and all my configuration so setting up a new machine is pretty straight forward. Moving the data back may not be as hard as I'm making it in my pea brain. It may be as simple as setting up a replication task from the backup to the Active NAS but I don't know.

Can someone give me a explanation of the restore steps with enough detail to walk an idiot through it.

Thanks a bunch for your help.

Tony T

 

[depasseg]

I would just replicate back to the original machine (either cli or a task) and then restore the config from backup (mainly because sometimes if you lose a dataset, the shares corresponding to that dataset are removed). And since there was a recent change to replication, you will need to make the datasets read/write instead of read only.

 

[Tony T]

I would just replicate back to the original machine (either cli or a task) and then restore the config from backup (mainly because sometimes if you lose a dataset, the shares corresponding to that dataset are removed). And since there was a recent change to replication, you will need to make the datasets read/write instead of read only.

Wouldn't you have to restore the config before you could replicate back to the original machine? How do you change the permissions on the datasets? I noticed in the GUI that I'm unable to change them.

Thanks,
Tony T

 

[depasseg]

You would need some config on the machine (IP address and SSH), but if your datasets weren't there yet, the restore wouldn't be effective.

There is a CLI option to change the datasets to read/write, but I don't know it off the top of my head.

 

[MrToddsFriends]

There is a CLI option to change the datasets to read/write, but I don't know it off the top of my head.

Johnny Fartpants suggested using

zfs set readonly=off tank/dataset

in this thread
https://forums.freenas.org/index.php?threads/replicated-datasets-are-now-readonly.41711/

but he seems to be unsure if this is the best way (and I am also).

 

[depasseg]

That will work (at least once), the concern was that it would be reset on the next replication.

 

[Tony T]

Thanks for the replies. I think the best course for our peace of mind is to actually build another 2950 box and do the restore live in our lab. Once I have it up and running I may need more help but I'll post the results and how we achieved them.

Thank you for your help.
Tony T

 

[cdiddy]

Thanks for the replies. I think the best course for our peace of mind is to actually build another 2950 box and do the restore live in our lab. Once I have it up and running I may need more help but I'll post the results and how we achieved them.

Thank you for your help.
Tony T

Hi Tony, did you ever do this? Did you post the results anywhere? I too am having a lot of trouble finding any details on what procedures to use to restore from a backup.

 

[Johnny Fartpants]

I think the first point to make is there are two types of disaster when it comes to your primary server failing. 1. After troubleshooting the issue is your pool lost. 2. After troubleshooting the issue is your pool good. Depending on how you answer those questions would depend on your next steps to recovery. Also when your primary system failed did you point users to your replica? and if so have they had read only access or read/write? If primary has lost its pool then you need to promote secondary to primary if you haven't already and depending on size of your pool, time and how angry people are that they can't write data you may choose to allow users read/write access to your secondary system thus it now becomes your primary. Once you have fixed the problem with your old primary and started again you can replicate back. Depending on the hardware differences between your primary and secondary servers you may choose to keep things as they are or you may wish to fail back to your original primary once replication has completed. This would probably involve uploading your original config file. If you primary pool isn't lost and you failed over to your secondary then it is possible to replicate back with incremental snapshots meaning you don't have to replicate the whole pool from scratch again. This is not straight forward and has a lot of gotchas but it is possible if you're careful. What you need to be very careful of however is if primary server goes down and users start to read/write to secondary if you fix old primary and bring it back online it will replicate and blow all the changes your users made on old secondary. So I would suggest you disable ssh on secondary before bringing primary back online and turn off snapshots and replication on old primary. I hope that makes some sense although it is very confusing. I have spent a lot of time going through these scenarios in practice on really pools so this isn't just what I think will happen its what actually happens.

 

[hokan]

I asked a somewhat similar question a few months ago, but I assumed that the secondary was to become the new primary and, once repaired, the old primary would be the new secondary. With a bit of help, I worked out a process that seemed to work.

Thread here: https://forums.freenas.org/index.ph...-a-replicant-to-be-primary.57219/#post-402601