How to back up databases and other data using FreeNAS as backup media

[lightwave]

I’m finally getting around to setting up backups for some of the services and databases that I’m running on my home network. The general idea is to run daily backups to a share on my main FreeNAS box and then replicate for a second and third backup copy. I realize there’s quite a few ways that this could be done, including whether and how to use separate files or snapshots to manage backup retention. I suspect I’m not the first one setting up something like this. I would therefore love to hear what you guys have tried, what you have learned, and what your setups look like today.

My main requirement is to have easy access to a number of backups of, say a database, stretching back in time. Ideally something similar to tape backups with a grandfather-father-son rotation scheme.

Some thoughts: One option would be to do a daily backup with the date in the file name and use a script to remove old copies that should no longer be retained. These backups would be really easy to access, but there would also be a risk of removing or corrupting all backups in one go. ZFS snapshots could be used to prevent direct altering, but would increase the number of backups stored (e.g. weekly snapshots kept for 10 weeks combined with daily backup files kept for one week practically means storing 10 weeks of backups at any one time). Another option would be to only keep one daily backup on the ZFS share and handle retention through daily snapshots. There is still a risk of removing snapshots by mistake, but at least it will not be possible for a virus to encrypt all backups in one go. How are you guys going about this?

For clarity: This thread is about backing up databases, GIT-repositories, Seafile libraries, e-mail accounts and other data using FreeNAS as the “backup media”, not about how to back up data already on a FreeNAS dataset. While interesting and relevant, the latter subject has been discussed extensively in other threads on the forum.

I look forward to hearing what you guys are doing.

 

[Heracles]

Hi,

Here, my services are provided by docker containers I run in a docker host from my ESXi server. In that Docker host, I created a container I named BackupAgent using a pretty basic Linux environment.

That one is equipped with everything needed to collect the data that must be backed up :
MySQLDump for the database
Read-only mapping of other containers' volumes
Should I need to get to other data to backup, I will configure the access from that Docker to wherever these data are.

That container also has an NFS mapping to a dataset in FreeNAS.

At the required frequency, the backup agent takes a backup, encrypt it using OpenSSL and then move it to the NFS share. I also put a timestamp in that directory (I output the date in a file named timestamp.txt).

That dataset is part of my 5 generations snapshot hierarchy :
15 minutes for 3 days
1 hour for 3 week
1 day for 3 months
1 week for 18 months
4 weeks for 4 years

These snapshots are then replicated using ZFS send / receive.

That way, no matter which backup I wish to use, I always have everything I need is one snapshot. Also, these backups are unusable as is from any of the servers on the receiving end. The backups being encrypted by the BackupAgent before being saved in FreeNAS, FreeNAS has only cryptograms and no keys.

For doing the restore, I have a RestoreAgent container on the other end. I pre-configured it with everything but the key required for decrypting the backups. That way, all I need to do a restore is to configure that key and then re-populate databases, docker volumes, etc.

 

[lightwave]

Hi Heracles,

Thank you for sharing the details of your setup.

We seem to have a pretty similar environment with ESXi virtualisation and services in docker containers. This makes your setup extra relevant as inspiration/input for my upcoming exercise. Do you mind if I ask a few follow-up questions around some of the details?

1. Does the backup agent remove/overwrite the files from the previous day?
2. Does the timestamp.txt file have any other purpose than to identify the date that the backup in a snapshot were taken (e.g. in the case that the agent failed)?
3. Do you use FreeNAS snapshot tasks to generate the snapshots? How do you name them?
4. Why did you chose to encrypt the backup files rather than just the ZFS pool?
5. What encryption algorithms do you use for the backups?

Again, thank you for sharing!

 

[Heracles]

1. Does the backup agent remove/overwrite the files from the previous day?

It overwrites them. That way, each snapshot contains only its own corresponding backups.

2. Does the timestamp.txt file have any other purpose than to identify the date that the backup in a snapshot were taken (e.g. in the case that the agent failed)?

Nope. It is just an easy way to confirm that the associated files present in the directory were taken by the script at that time. Should the script fails for any reason and not take any backup, that file would tell me that the backups are not the right one and when they are dated back.

3. Do you use FreeNAS snapshot tasks to generate the snapshots? How do you name them?

Yep. I let FreeNAS names them automatically. They end up being something like PoolName/DatasetName@auto-YearMonthDayHourMin-retention

4. Why did you chose to encrypt the backup files rather than just the ZFS pool?

Because ZFS encryption is almost useless plus it is extremely dangerous. This forum is full of people who lost their pool and data thanks to ZFS encryption. Second, if ZFS has both the cryptograms and the keys, it can recover cleartext data easily. And that is what it does as long as your FreeNAS server is up and running.

By encrypting outside ZFS, both my pool and data are much safer and I do not risk to loose it all to that self-inflicted ransomware. Also, because the key never makes it to FreeNAS, none of my FreeNAS server can recover the data by itself. Should a server be stolen, accessed physically or fully compromised, data will remain safe.

5. What encryption algorithms do you use for the backups?

The tool I used is OpenSSL and I configured it to use AES-256-CBC.

My pleasure to share information with you and congrats for planning a good and real backup for your data. So many here lost their data because they did not had any backups... Just be sure that you will do restore tests after you took your backups. A backup that has not been restored is worthless.

Even here, I failed my last yearly restore test. I found out why, fixed the backup, re-did the test on the spot and did a second half-year restore test last month. Without my yearly test, I may well have found that my backups were corrupted only on the very day that I would have need them.

 

[lightwave]

Thanks again!

I can only chime in on the dangers of using the current encryption support in FreeNAS. It basically feels like a thousand ways to shoot yourself in both feet simultaneously. That said, I actually do use GELI-encrypted disks for my current pool and with some precautions i feel reasonably safe that I will not lose all of my data. That might, however, be the subject of another thread.

On a very nerdy cryptography note: If you are really paranoid about data security, you could take your setup two steps further: (1) by using AES-256-GCM in place of CBC you would get cryptographic authentication of the data as well (to discover errors/changes in the cipher stream) and (2) by generating new AES keys for each daily backup and encrypting these keys using public key cryptography (RSA or ECC) you could avoid storing any sensitive keys in the docker containers until they are needed for recovery.

And thanks for the reminder of the importance of doing recovery tests... I think we all need to hear that once in a while

 

[lightwave]

I'd love to here about more cool backup solutions involving FreeNAS as the backup media, if any of you have the time to share your setups ...

 

[danb35]

Because ZFS encryption is almost useless plus it is extremely dangerous.

Possibly picking at nits, but it seems worth pointing out that FreeNAS isn't using ZFS encryption; it's using its own form of GELI encryption to encrypt the disks, on which it creates a ZFS pool. ZFS encryption (which will allow individual datasets to be encrypted if desired, rather than the entire pool) will land with TrueNAS 12. I'm not sure how well-characterized either its safety or its security are.

 

[lightwave]

Possibly picking at nits, but it seems worth pointing out that FreeNAS isn't using ZFS encryption; it's using its own form of GELI encryption to encrypt the disks, on which it creates a ZFS pool. ZFS encryption (which will allow individual datasets to be encrypted if desired, rather than the entire pool) will land with TrueNAS 12. I'm not sure how well-characterized either its safety or its security are.

Thanks for the clarification @danb35. I did refer to the GELI-based disk encryption available through FreeNAS user interface and I'm pretty sure that was what @Heracles was talking about as well.