Full off-site sneakernet backup solution?

[xhoy]

Dear freenas comunity,

We are happy freenas users for some time. I was looking at a solution to backup our freenas system. And i was unable to find anything that was workable.

I most wonderd that the word “backup” wasn’t even mentioned in the docs.

I think a good backup should check these three boxes

1. must be offsite
2. must be on different media (hdd, tape, cd/dvd etc)
3. must contain different versions of the data (going back in time is essential)
   

Our system is only used for 20+ vm’s (nfs mounts) with a total size of ~5TB (total volume size is 8TB). Since we cannot create a offsite backup over the internet (poor internet speeds) we are forced to use sneakernet.

Then there are the requirements from our side:

1. Must use sneakernet
   
2. Low impact on performance (prefered to run at night or secondary system)
3. Daily notifications of status (backup was successful, backup failed...)
4. Must be low on daily maintenance (only changing disks, tapes or what ever)
5. Must contain multiple versions of the system (1 year, last 12 months, last 4 weeks, last 12 days..)
6. Should cost no more than $2K?
7. May be difficult to implement or time consuming (one time effort)
   

The only answer i found was attaching some drivers (esata/usb) and zfs send some snapshots over. But every thread that mentioned this noticed that swapping disks may cause issues. and splitting volumes isn’t possible (afaik).

Disclamer: is really tried searching the forum and google found SOME solution but not that everybody agreed on being “good” or not realy failure error-prone.


I would love to hear your ideas.

 

[Lix]

What kind of ISP speed are we talking about? What kind of hypervisor are you using?

 

[xhoy]

We have three esxi servers and one freenas. Internet speed is <10mbit upload and we generate about 100gb's a week of delta's. This would result in 25+ hour's of transfers. 5 hours every night. And this is average so when we do a larger migration it would easily generate more 20GB a day.
And we need that internet for other applications as well :) so we could at max utilize maby 5mbit, resulting in a 10 hour transfer window...

 

[mpfusion]

I'll explain how we do our backups. You possibly could adapt some ideas to implement your own solution.

I think a good backup should check these three boxes
must be offsite
must be on different media (hdd, tape, cd/dvd etc)
must contain different versions of the data (going back in time is essential

We use 5.4TiB HDDs in external eSATA bays. So you can just pop in/out a disk and you're done. The disks are labelled, so we and the system knows which disk is connected.

Our backup disks used to be formatted with ext3. But we recently switched to BTRFS because the pool size is expected to outgrow the size of a single disk soon. And you get transparent compression as well.

We could surely have used zfs as well. But I'd rather like our backup system reside on a different file system.

Our system is only used for 20+ vm’s (nfs mounts) with a total size of ~5TB (total volume size is 8TB).

This should fit on a single disk. When you use compression you'll have some more headroom.

Since we cannot create a offsite backup over the internet (poor internet speeds) we are forced to use sneakernet.

Same here.

Then there are the requirements from our side:
Must use sneakernet
Low impact on performance (prefered to run at night or secondary system)
Daily notifications of status (backup was successful, backup failed...)
Must be low on daily maintenance (only changing disks, tapes or what ever)
Must contain multiple versions of the system (1 year, last 12 months, last 4 weeks, last 12 days..)
Should cost no more than $2K?
May be difficult to implement or time consuming (one time effort)
The only answer i found was attaching some drivers (esata/usb) and zfs send some snapshots over. But every thread that mentioned this noticed that swapping disks may cause issues. and splitting volumes isn’t possible (afaik).

We use a different computer (not the FreeNAS server) running Debian and call a custom script via cron which then mounts the corresponding disk connected via eSATA and then calls rsnapshot. This creates a hard link farm including a configurable amount of history using rsync and SSH behind the scenes.

So basically we rely on rsnapshot for the backup and add some scripting to completely automate it. If some error occurs we'll receive an email because that's what cron does when a command generates output. You can run this cron job e.g. once per week at night.

The cost for us was one 5.4TiB disk (WD green) and an oldish spare computer → approx. $300

If you want a more elegant solution you could use ZFS on the backup host and just do a zfs send/receive. That's way more efficient (especially for zvols), much faster and it saves space as well.

This system is in place since several years and proved to work reliably. Additionally we do have an offsite FreeNAS server that we use as replication target.

 

[Lix]

How many CPU sockets per server? I use Veeam you see.. Depending on what features you need and the amount of sockets your hypervisors have, you might get come close to a 2K investment.


To answer per points in your initial post:


1. Veeam can seed backups. Meaning, backup to USB drive and take that to your DR-site and merge with the job from your production site.

2. First run could impact performance, but you can also throttle backup based on different parameters like limit speed to USB by a set sum.

3. Nice email reports from the Veeam server.

4. I rarely touch the system… Depends on how dynamic your environment is I guess J

5. Veeam has a feature for this.

6. This depends on your needs..

7. Easy I would say


Your target could be smaller FreeNAS box J I have not take into account windows licenses.


Using ZFS snapshots could also work, setup a backup FreeNAS at production site and move it to DR-site after initial replication. I will let the more experienced FreeNAS people talk about this alternative.

 

[xhoy]

Thank you for you reply.

So basically we rely on rsnapshot for the backup and add some scripting to completely automate it. If some error occurs we'll receive an email because that's what cron does when a command generates output. You can run this cron job e.g. once per week at night.

The option you are supplying crossed my mind but when a VMDK changes rsnapshot transfers the hole file (again). this would result in syncing 5TB every night.....

Edit: I do like the BTRFS suggestion.... would it be possible to zfs send a freenas/freebsd volume to a debian machine? and then rsnapshot that to a external drive? or do we need three machines then... going to look in to that.

 

[xhoy]

How many CPU sockets per server? I use Veeam you see.. Depending on what features you need and the amount of sockets your hypervisors have, you might get come close to a 2K investment.

Veeam is out of question since we then lose our snapshot ability on the esxi side. since veeam cannot backup machines that have snapshots :'(
and we use those a lot.

 

[mpfusion]

The option you are supplying crossed my mind but when a VMDK changes rsnapshot transfers the hole file (again). this would result in syncing 5TB every night.....

That's the case, yes. We do mainly use NFS for file storage and NFS root mounts for diskless clients. So it's all file based. The few VMs we have (on zvols) are replicated once a week. But that's ok for us. It might not be the best fit for you.

Edit: I do like the BTRFS suggestion....

I don't know if I like it myself. I felt more comfortable with ext3, but it doesn't suit our needs any more.

would it be possible to zfs send a freenas/freebsd volume to a debian machine? and then rsnapshot that to a external drive?

That's how we do it for zvols. We store the zfs send stream in a file and rsnapshot picks it up. However, as you have already noticed rsnapshot is not the best tool for large files which change regularly.

Maybe it's better for you to have a dedicated host (e.g. Debian or FreeBSD) and just use zfs send/receive and attach the disks using hot swap enclosures (eSATA, SATA, etc.). You'll get a much smaller daily diff that way. But of course you'd have to run ZFS on the host in that case. For you that seems to be the better option.

 

[xhoy]

That's the case, yes. We do mainly use NFS for file storage and NFS root mounts for diskless clients. So it's all file based. The few VMs we have (on zvols) are replicated once a week. But that's ok for us. It might not be the best fit for you.

Yeah i noticed that most people do it this way, that is way i wondered how this would work out in our case.

I don't know if I like it myself. I felt more comfortable with ext3, but it doesn't suit our needs any more.

Proven tech is always better :)

That's how we do it for zvols. We store the zfs send stream in a file and rsnapshot picks it up. However, as you have already noticed rsnapshot is not the best tool for large files which change regularly.

So I use my freenas and sync it to a debian machine? Didn't know zfs was cross platform!

Maybe it's better for you to have a dedicated host (e.g. Debian or FreeBSD) and just use zfs send/receive and attach the disks using hot swap enclosures (eSATA, SATA, etc.). You'll get a much smaller daily diff that way. But of course you'd have to run ZFS on the host in that case. For you that seems to be the better option.

So you suggest that(example):

1. buy $amount 8TB archive hdd's (example)
2. run $OS
   
3. ZFS send/receive diffs for Freenas
4. ZFS send/receive diffs to external HDD's

If we would like to keep some amount of versions there is a real possiblity that we need to split the backup's to several disks, i cant find anything on zfs being capable of doing that? So step 4 would need to be btrfs?

Or could we rsync the snapshots separately? so we would have 1 masterdisk + diff-disk?

 

[mpfusion]

So I use my freenas and sync it to a debian machine? Didn't know zfs was cross platform!

Sort of. You can run ZFS on numerous operating systems. But the supported pool features vary. And I don't know if it's possible to zfs send a snapshot on a current FreeNAS system and zfs receive it on a Debian or FreeBSD system. If you main pool doesn't use features that aren't available on the receiving side it should work without issues.

Using ZFS/BTRFS you have the following advantages:

• more efficient than e.g. rsync
• your backup pool can easily span multiple disks, so no need to manually split the data
• snapshots can easily be retained for X days, weeks, etc.

If we would like to keep some
amount of versions there is a real possiblity that we need to split the
backup's to several disks, i cant find anything on zfs being capable of doing
that? So step 4 would need to be btrfs?

ZFS and BTRFS can do that. You could have a backup pool of e.g. 1x6TiB disk or 6x3TiB in RAIDZ2, etc. No need to manually distribute the data.

Or could we rsync the snapshots separately?

That's possible and that's basically what we do here. But it doesn't seem to be a good fit for you. When you start storing incrementals you cannot ever delete the original snapshot because the incrementals depend on them. So every once in a while you need to transfer a new master snapshot.

What I would do in your situation:

Get a dedicated server. Find an OS that runs ZFS and is capable of receiving the snapshots. Probably I'd try a current FreeBSD. In might be that the ZFS version of FreeNAS might be to new even for a current FreeBSD, but I'm not sure. FreeNAS will work of course. But it's an appliance and I'm not sure if it's a good idea to tinker with the system. Therefore I'd rather use plain FreeBSD if possible.

Attach the desired number of disks to the host so you can easily remove them (hot-swap bays, external eSATA enclosure, etc.). Create a script that first imports the pool, then transfers the snapshot from the main system to the local pool, then exports the pool again. The disks can now safely be removed.

So in practise you only need to plug the disks and the backup runs automatically and you can safely remove the disk(s) afterwards and take them home. Instead of using cron to trigger the backups, inserting the backup disk could also be used as a trigger event to start the backup. But that's for you to decide and implement.

There are many other ways to to the backup. Manual solutions like the one described here, larger entreprise solutions (e.g. Bacula or Bareos which are very efficient at handling incrementals, but intended for larger installations) or anything in between.