Off-site Replica/Backup

[lucapsg]

Hi all,
I have been testing the snapshot replication between two FreeNAS systems for a few days and it seems to be working very well!
Now I'd like to send snapshots to off-site cold storage and possibly manage their rotation.
The goal is to reduce recovery times even in the event of a complete disaster, simply by downloading the data onto a new machine.
We speak at most of 3TB.

Now the questions ... ;-)
1. Since it is possible to save snapshots as a file, does someone already send them to cold cloud storage?
1a. Is there any way to manage its rotation?
1b. How to restore on a newly installed system?
1c. Is there any tool/plugin in the FreeNAS GUI that does it, without having to use the command line?
2. In addition to the various rsync/scp/... what other tools/protocols can be used to obtain something similar, capable of maintaining N versions of the data?
2a. Would it be able to maintain file and folder permissions?
3. Which off-site storage provider do you recommend in Europe?

Thank you.

 

[Heracles]

Hey Luca,

I am doing something like that myself. My 3 servers are in my signature...

I configured the first server (Atlas) and put all my data in.
While on site, I configured server No 2 (Hades) and did a complete snapshot replication task to over LAN.
Once sync to Hades completed, I moved it offsite and put it back online there.
I modified the DNS setting for pointing to Hades' new IP address.
Atlas found it back and re-sync it with only the few and small last snapshots.
Since then, Hades is kept offsite and in sync with Atlas all the time.

Back on site, I configured Thanatos and plugged it in a dedicated NIC in Atlas
I did a first complete replication to it.
I then powered it off and I keep it off most of the time.
About once per 2 weeks and never less than once a month, I power On Thanatos
Once online, Atlas finds it back and update it.
Once Atlas confirmed that Thanatos is up to date, I power it back off.

That way, I have an always up-to-date copy on Hades. Should Atlas suffers physical damage or logical damage that does not propagate, Hades has everything up to the last 15 minutes.
Should something happens to Hades, no problem. It is not the main server; Atlas is.
Should something VERY wrong happen to Atlas and propagate to Hades, Thanatos is safe because it is offline. In that case, all I need is to wipe Atlas, re-deploy it and re-sync it from Thanatos.

ZFS replication will not only send the newest snapshot, but expired ones will also get removed.

This propagate file ownership, permissions, etc. It is also easy to save a copy of your config in the pool and have it sync to the other servers.

Thanks to that, I have my 3 copies all on my own servers.

 

[Basil Hendroff]

The User Guide (UG) is your friend.

The goal is to reduce recovery times even in the event of a complete disaster, simply by downloading the data onto a new machine.
We speak at most of 3TB.

3. Which off-site storage provider do you recommend in Europe?

Have you considered a Cloud Sync option? It may be cheaper, simpler and quicker to access than a physical location.

1. Since it is possible to save snapshots as a file, does someone already send them to cold cloud storage?

Is snapshotting really a consideration in this case, if all you want to do in a complete disaster is simply download the data to a new machine?

1b. How to restore on a newly installed system?

I like to think of snapshots as a way of versioning and replication as a means of near-real-time pool/dataset duplication. With that in mind, if minimum downtime is a KPI, then Importing a Pool from a nearby replication target may be the quickest way to bring a system back online.

1a. Is there any way to manage its rotation?

Have a look at Destination Snapshot Lifetime in the UG under Replication.

1c. Is there any tool/plugin in the FreeNAS GUI that does it, without having to use the command line?

It's built-in and readily accessible through the GUI. There is no requirement to use the command line unless you want to.

2. In addition to the various rsync/scp/... what other tools/protocols can be used to obtain something similar, capable of maintaining N versions of the data?

If your client base use Windows PCs, have a look in the UG on Configuring Shadow Copies, which rely on snapshots. It becomes a trivial exercise for users to restore earlier versions of their own files.

2a. Would it be able to maintain file and folder permissions?

Yes it does.

 

[Basil Hendroff]

Back on site, I configured Thanatos and plugged it in a dedicated NIC in Atlas
I did a first complete replication to it.
I then powered it off and I keep it off most of the time.
About once per 2 weeks and never less than once a month, I power On Thanatos
Once online, Atlas finds it back and update it.
Once Atlas confirmed that Thanatos is up to date, I power it back off.

@Heracles My data risk appetite is such that I've implemented two out of three of the 3-copies rule that you've described. What I'm missing is copy #3 - the offline copy. I'm curious to know whether you would consider a manual cloud sync of storage data, which is then disabled, the equivalent of copy #3?

 

[Heracles]

Hi,

whether you would consider a manual cloud sync of storage data, which is then disabled, the equivalent of copy #3?

What is important for the 3rd copy to be considered as offline is that no logical incident should be enough to reach it. Here, if you configure your cloud credentials in your server, a logical incident in the server can reach these credentials and propagate to the cloud. As such, I would say that No, such a setting is not immune against logical access, so is not completely offline, even if you trigger the sync manually. I would consider that as a 2nd version or 2nd type of copy No2.

Should the requirements for the cloud access never be in the server, like a TOTP code you enter manually every time and revoke once the sync is done, then you can consider that close enough. No logical incident can grab that TOTP and re-use it, so you can consider it as offline.

What you could do is to use that cloud sync as your second copy and keep it sync all the time. Then you would need something like my Thanatos server, an old piece of crap that can barely run the storage you need. You then deploy that third storage at your place and sync it once in a while, powering it down between 2 syncs. Then you would have your first copy, the cloud as a 2nd copy that is online and offsite and a third copy that is offline.

 

[Adrian]

I currently back up some 400 GB "critical" data to AWS S3.
I am toying with the idea of switching to rsync.net, using rsync which would cost about the same in storage (around $10 a month) but with no egress fee (Amazon's is $0.09 per GB).

Backing up my "important" data (around 3.5 TB) with rsync.net using zfs send fits quite nicely with rsync.net's minimum 4 TB's worth minimum charge. But at $100 month that is starting to get expensive, especially as my current off-site partially replicated server is an old FreeNAS Mini with a replacement motherboard.

Including my less well backed up "hazarded" data I would be a bit under 10 TB, at say $250 a month. You can rent / buy & depreciate quite a lot of real storage for that!

Another issue for me is bandwidth. I am on a good 80 / 20 (Mb) VDSL2 connection. Uploading terabytes takes a long time, downloading 4 times that. Gigabit FTTP might be available in a few years, but will be unaffordably expensive.

So, for me high volume cloud storage is a pipe dream.

Meandering back to Luca's questions, rsync.net and rsync might be the way to go.

 

[Basil Hendroff]

Including my less well backed up "hazarded" data I would be a bit under 10 TB, at say $250 a month. You can rent / buy & depreciate quite a lot of real storage for that!

So, for me high volume cloud storage is a pipe dream.

Yes, you're right on both counts. I'd forgotten about that.

 

[lucapsg]

@Heracles , if you have a second place in which to put another box, yours is definitely an excellent solution.

Unfortunately I don't have it and therefore I'm trying to understand if there is a (practical) way to send and manage snapshots in an off-site storage and maybe get a functionality similar to the FreeNAS Replication Task, including management of the versions.

Meandering back to Luca's questions, rsync.net and rsync might be the way to go.

I took a look at Rsync.net and it's actually very interesting. Unfortunately with rsync I would probably lose the permissions of SMB shares.
I also don't have a great connectivity but at least the amount of data changed every day is rather limited (10GB).

I think that saving snapshots in the form of files in cold storage and managing their rotation from the source system could give significant advantages compared to rsync, at least in a DR scenario:
- sending a snapshot is efficient
- freely selectable number and frequency of versions
- identical recovery, including file permissions (!)
- wide choice of suppliers
- the price of a "dumb" storage should be lower

By the way, I don't think it's so complex to do.
And maybe someone already does ...
Step forward ;-)

 

[enaugavule1]

So snapshot is possible.
can we TRUENAS on hyper-v and do a offsite replciation backup on another virtual machine or physical machine?

 

[Heracles]

Hey @enaugavule1,

So snapshot is possible.

Clearly.... But I do not see how this is related to the post...

can we TRUENAS on hyper-v and do a offsite replciation backup on another virtual machine or physical machine?

ZFS send / receive can target either : a physical or virtual destination. Be extra careful when doing virtual and never use logical or virtual drives with TrueNAS, but if you did it correctly, you can sure ZFS send to that VM. Also, know that the only hypervisor with actual experience and runtime for TrueNAS is VMWare.

But here again, how is that related to the post ?