Backup changes without having the full Dataset on Backup drive

[TobiT92]

Hi guys,

first of all, I am happy with a link as a reply, that heads me in the right direction. I tried to search for my specific question but didn't get a great result, mainly because I do not know what terms to search for other than "backup changed files only" which does give me a lot of solutions like rsync etc but they all share the same problem: They need to have the full dataset available on the backup to know what files are new etc.

What I am looking for is a way to let a backup solution look at a specific dataset or a specific folder, remember the current status, and then in future backups ONLY transfer changed files to an external drive or another pool that does not contain the full dataset or the whole folder.

Why do I need that? I have a freenas server in the office that currently has over 14tb of data. At my home, I will have another freenas server, with the same data (copied over, replicated, whatever) as a backup.

Now if I create or change files in my office, I want to get the changed files to my server at home. I know this is possible directly over the internet, however, I have concerns in terms of transfer speed (10mbit upload) and security (opening up my servers to the internet).

What I rather want to do is plug in a harddrive into my office server, get all the changed files, take that home, and copy those files over.
As there are no harddrives bigger than 14tb currently, I cannot have the full dataset on this transfer-drive. The server storage will grow to 20tb in the next months probably, so this is not feasible.

So, is there a way to make just an index of the current dataset that can then be used to compare what files need to be copied to the transfer drive?


Thanks!
Tobi

 

[melloa]

What I do is use a VPN tunnel. Two small boxes with pfSense and a site-to-site tunnel. That way you can have rsync or another method running without exposing the servers to the internet. Think about this if you don't get the solution you are looking for.

 

[TobiT92]

Thanks melloa, can you hint me to a guide on how to establish such a VPN Tunnel? My cable modem that connects me to the internet at home only has ipv6 port forwarding, is that a problem?

 

[melloa]

can you hint me to a guide on how to establish such a VPN Tunnel?

As I've mentioned, I do it with pfSense. Here's a guide from their site, but there are hundred guides out there that you can find with a search engine.

My cable modem that connects me to the internet at home only has ipv6 port forwarding, is that a problem?

Not to my knowledge, as pfSense handled IPv4 and IPv6.

If you are successful with your tunnel, I'd take the back-up server to the main site, run a local sync, so only the changes after that will be done via internet.

 

[sretalla]

If you still want to go down the original road you were asking about (I agree it's much better to use a VPN and just let it go over the net), you could use rsync with a time modifier (you would need to manually set that based on the last copy you did)

rsync -RDa0P --files-from=<(find sourcedir/./ -mtime -7 -print0) targetdir/

Where targetdir/ is a location on your backup/transfer disk, sourcedir/ is the server root (or the root of the file location you're interested in) and -mtime -7 in the example means all files modified in the past 7 days.