I'm running I Freenas 11.2.u4 box and i have a high speed access pool of all ssd's and then i have the spinning rust drive pool for back up storage. I can't seam to figure out how to set up Rsync to copy everything from my high speed pool to the spinning rust pool for auto backups nightly. right now i have to use a program on one of my windows computer that syncs the pools. But I'm looking for a way that i can do that backup and if rebuild with out it being over a gigabit line if i need to rebuild a pool for any reason. Everything i find shows 2 different boxes. If anyone can point me in the right direction or help me out that would be awesome.
-
Important Announcement for The TrueNAS Community.
The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
Rsync same Box
- Thread starter oumpa31
- Start date
ethereal
Guru
- Joined
- Sep 10, 2012
- Messages
- 762
i use a couple of scripts that get run as a cron job. i don't know if this will help
rsync -rltgoDvs --backup --backup-dir="/mnt/Storage/First Backup/Photography/Changes/" \
--suffix=_`date +%Y-%m-%d_%H_%M` "/mnt/Working/Photography/Photographs/" \
"/mnt/Storage/First Backup/Photography/"
rsync -rltgoDvs --backup --backup-dir="/mnt/Storage/First Backup/Photography/Changes/" \
--suffix=_`date +%Y-%m-%d_%H_%M` "/mnt/Working/Photography/Photographs/" \
"/mnt/Storage/First Backup/Photography/"
FJep
Dabbler
- Joined
- Mar 7, 2019
- Messages
- 38
I use rsync to copy my music and photos on my zvol (VolZ1) to another single disk (VolA1).
Keep in mind, it's not a real backup, since accidental deletions will be synced to your copy. I keep a copy on an external disk.
Here is how to set it up.
First you have to create a rsync module like this:
Then create a rsync task like this:
Keep in mind, it's not a real backup, since accidental deletions will be synced to your copy. I keep a copy on an external disk.
Here is how to set it up.
First you have to create a rsync module like this:
Then create a rsync task like this:
FJep
Dabbler
- Joined
- Mar 7, 2019
- Messages
- 38
If you followed my instructions and added extra options with a directory and naming pattern for a log file you will have a log file where you can see what was copied and how long it had taken. My backup location is also a share. So I can check it from my Windows PC.
My rysyncs of my music (300GB) and photos (1000GB) need around 40s if nothing changed plus some time if copies had to be made.
My rysyncs of my music (300GB) and photos (1000GB) need around 40s if nothing changed plus some time if copies had to be made.
Why not use periodic snapshots and replication since you are copying the whole pool?
It's instant (no scanning of files) and works like a charm.
An advantage is that you only send the increment changes, AND you can mount any snapshot you have.
So if you do daily snapshots, weekly and monthly (each one keeping for a specific time) you can have many points in time you can go back to simply by mounting the snapshot.
All of this can be set up in the gui under Tasks \ periodic snapshot and Task\replication
Under replication provide the local host as the target and the spinning rust pool as the location to back up to.
It should be noted that the only danger is that if you have large amount of data that is MODIFIED (not added) every day, the size of all snapshots together can become a lot larger than the total size of the dataset.
This you can test however by not keeping the snapshot for a too long time in the beginning. If you see that the total size of your snapshots doesn't explode you should be fine to increase the retention time.
For reference, I'm snapshotting:
every hour (keep a day) while the server is being actively worked on. No performance hit is noticeable.
every day(keep 7 days)
every week (keep 5 weeks)
every month (keep half a year)
and that on about 2 TB of data without issues.
It's instant (no scanning of files) and works like a charm.
An advantage is that you only send the increment changes, AND you can mount any snapshot you have.
So if you do daily snapshots, weekly and monthly (each one keeping for a specific time) you can have many points in time you can go back to simply by mounting the snapshot.
All of this can be set up in the gui under Tasks \ periodic snapshot and Task\replication
Under replication provide the local host as the target and the spinning rust pool as the location to back up to.
It should be noted that the only danger is that if you have large amount of data that is MODIFIED (not added) every day, the size of all snapshots together can become a lot larger than the total size of the dataset.
This you can test however by not keeping the snapshot for a too long time in the beginning. If you see that the total size of your snapshots doesn't explode you should be fine to increase the retention time.
For reference, I'm snapshotting:
every hour (keep a day) while the server is being actively worked on. No performance hit is noticeable.
every day(keep 7 days)
every week (keep 5 weeks)
every month (keep half a year)
and that on about 2 TB of data without issues.
I let it run over night and it looks like everything was copied over 8.96TB of data. I probably started it around 9 pm last night and i checked it when i got up at 8 am. I'm not exactly sure how long it ran because i forgot to set up the log system.
I now have it set up like this.
--log-file=/mnt/Vault/log/rsync_Zeus-'date +\%Y\%m\%d'.log
I do have snap shots but its mostly when i go to do an update to the system. I like to just have an exact copy of everything just encase something happens.
I started looking into zfs send | zfs receive just encase i need to rebuild one of the pools quickly but it seams like it may be more of a hassle.
I now have it set up like this.
--log-file=/mnt/Vault/log/rsync_Zeus-'date +\%Y\%m\%d'.log
I do have snap shots but its mostly when i go to do an update to the system. I like to just have an exact copy of everything just encase something happens.
I started looking into zfs send | zfs receive just encase i need to rebuild one of the pools quickly but it seams like it may be more of a hassle.
It's realy easy. In fact so easy that I'll try to make a step by step using only the gui as it is fully possible to configure it "set it and forget it" without the command line. I'm on my phone right now but I think I'll have some time tonight.
Snapshots and replication is a realy powrfull function of zfs so I at least want to share how it works as it has saved my *** many times.
Thanks a ton every guide I've found on zfs send | zfs receive was pre 11.2 so they all used the shell. I don't mind using the shell but i can just see something going wrong and loosing 1000's of movies, tv shows, music and family pictures. Wife would kill me if i lost them.
Ok here we go....
For this demo I should show you what I'm intending to do.
Let's say my "W" pool is your ssd pool and D is your spinning rust pool. for my demo I'll snapshot "Shares" and replicate it to my D pool (note that there is no shares dataset on my D pool)
The first step to get things going is to create an periodic snapshot task. You can find this under the menu 'Tasks'
Click the add button and you will be greeted with a screen you have to fill in.
I've filled in my form as follows. I'll explain below.
1) Pool/Dataset is the source data. So in this example I selected W/Shares. This can be selected from the dropdown list.
2)Recursive is kind of self explanatory, but it makes a snapshot of datasets under the source pool/dataset. So in my case if I would choise my pool W as a the pool, and than select recursive a snapshot would be made of all the datasets under the pool W. Also if you later add a new dataset in this pool.
3) Snapshot lifetime is how long you want to keep the snapshot. This means that the snapshots that are created on the source pool will be kept fot this amount of time.
4) Interval is self explanatory. How often do you want to make a backup. However it's important that the first snapshot you define has a long interval. Long enough to copy all data to the new pool before the next snapshot will start. So I would propose to set this to a week for now and set the liftetime to a month. Once everything is set up and the first replication has competed you can add as many schedules you wish. (weekly dayly hourly, each with their own lifetime.)
Make sure you set begin, end and days of the week in such a way that a snapshot will be created in the near future. Otherwise you have to wait for a long time before the first snapshot is made, and thus before the first replication will hapen in the next steps
5) days of week... at what days do you want to make snapshot..... standard is weekdays so you might want to add saturday and sunday.
6) enabled is chedked as default. I would propose to keep it that way :P
click save.
Now go to Tasks\replication tasks.
and select add
Fill in the form like this:
Pool dataset will be the pool you made the periodic snapshot task for. If you did not do this, you can not select the pool/dataset from the dropdown list.
Remote ZFS Pool/Dataset is the pool you want to send the data to. It says remote because you can send it to another host or server. In your case the host will just be "localhost" so select the location you want the replicated data to be send to (your spinning rust pool).
Check all check marks.
You don't need to change begin time or end time. As soon as a snapshot is made it will start to replicate. So you have full controll over when you want the replication to start and end by just scheduling your snapshots.
click scan ssh key (this will copy the ssh key of "localhost" to the proper location so the system trusts itself :))
Click save and you are done...
That's it!!!
It's a long description but honestly now you know where the buttons are you probaby can do it in 2 minutes flat.
now give your system time to make a snapshot (depends on your schedule) and to replicate (depends on how big your dataset is)
after this you can go back to storage\pools and you will see the dataset moved to your spinning rust pool as you can see here under.
Now you can add as many periodic snapshot tasks for your ssd pool as you want with the lifetimes you want to have and your spinning rust pool will stay up to date with your snapshots.'
Since zfs knows what blocks have changed since the last snapshot this method is "snappy" and only the changes are send over. Also because you will have many snapshots you can go back to any point in time for which you have a snapshot. This is very usefull if you deleted that one file that you want to restore. Just mount the snapshot, copy the file over to your live dataset and unmount the snapshot.
If you want to have an explination about how to do that, please let me know. I'll do another writeup like this.
I hope this helps!!!
For this demo I should show you what I'm intending to do.
Let's say my "W" pool is your ssd pool and D is your spinning rust pool. for my demo I'll snapshot "Shares" and replicate it to my D pool (note that there is no shares dataset on my D pool)
The first step to get things going is to create an periodic snapshot task. You can find this under the menu 'Tasks'
Click the add button and you will be greeted with a screen you have to fill in.
I've filled in my form as follows. I'll explain below.
1) Pool/Dataset is the source data. So in this example I selected W/Shares. This can be selected from the dropdown list.
2)Recursive is kind of self explanatory, but it makes a snapshot of datasets under the source pool/dataset. So in my case if I would choise my pool W as a the pool, and than select recursive a snapshot would be made of all the datasets under the pool W. Also if you later add a new dataset in this pool.
3) Snapshot lifetime is how long you want to keep the snapshot. This means that the snapshots that are created on the source pool will be kept fot this amount of time.
4) Interval is self explanatory. How often do you want to make a backup. However it's important that the first snapshot you define has a long interval. Long enough to copy all data to the new pool before the next snapshot will start. So I would propose to set this to a week for now and set the liftetime to a month. Once everything is set up and the first replication has competed you can add as many schedules you wish. (weekly dayly hourly, each with their own lifetime.)
Make sure you set begin, end and days of the week in such a way that a snapshot will be created in the near future. Otherwise you have to wait for a long time before the first snapshot is made, and thus before the first replication will hapen in the next steps
5) days of week... at what days do you want to make snapshot..... standard is weekdays so you might want to add saturday and sunday.
6) enabled is chedked as default. I would propose to keep it that way :P
click save.
Now go to Tasks\replication tasks.
and select add
Fill in the form like this:
Pool dataset will be the pool you made the periodic snapshot task for. If you did not do this, you can not select the pool/dataset from the dropdown list.
Remote ZFS Pool/Dataset is the pool you want to send the data to. It says remote because you can send it to another host or server. In your case the host will just be "localhost" so select the location you want the replicated data to be send to (your spinning rust pool).
Check all check marks.
You don't need to change begin time or end time. As soon as a snapshot is made it will start to replicate. So you have full controll over when you want the replication to start and end by just scheduling your snapshots.
click scan ssh key (this will copy the ssh key of "localhost" to the proper location so the system trusts itself :))
Click save and you are done...
That's it!!!
It's a long description but honestly now you know where the buttons are you probaby can do it in 2 minutes flat.
now give your system time to make a snapshot (depends on your schedule) and to replicate (depends on how big your dataset is)
after this you can go back to storage\pools and you will see the dataset moved to your spinning rust pool as you can see here under.
Now you can add as many periodic snapshot tasks for your ssd pool as you want with the lifetimes you want to have and your spinning rust pool will stay up to date with your snapshots.'
Since zfs knows what blocks have changed since the last snapshot this method is "snappy" and only the changes are send over. Also because you will have many snapshots you can go back to any point in time for which you have a snapshot. This is very usefull if you deleted that one file that you want to restore. Just mount the snapshot, copy the file over to your live dataset and unmount the snapshot.
If you want to have an explination about how to do that, please let me know. I'll do another writeup like this.
I hope this helps!!!
please note that I used snapshot and backup kind of interchangingly. A snapshot is not a backup. A backup should be on 2 different types of media, and on two phisically different locations. But it's better than hoping your ssd drives will not run out.
Thanks that is a huge help actually. My SSD pool is 8.96tb currently. I know that is a lot of data but any idea how long it would take to replicate everything to the spinning rust.
Once its replicated i would be able to the files just like they were on the ssd's if i mapped my computer to them correct?
Once its replicated i would be able to the files just like they were on the ssd's if i mapped my computer to them correct?
Not exactly.
Keep in mind you are potetially making tens of snapshots per day (depending on your snaphot schedules).
They are not automatically mounted. You have to go to a specific snapshots and mount and then share that mounted snaphot.
I'll show you how tomorrow if you are interested. Now it's time for me to turn of my brain.
Let me know if you would like me to do that.
asuming your hd's are kind of modern and can maintain about 50 MB per second (thats conservatively) you should be able to replicate in about 9*1024*1024 / 50 MB/s = 188 743 seconds or 52 hours. My disks do about 160 MB/s so this could be a lot less..... but it's better to keep this in mind when you schedule your first snapshot.
9TB is a LOT of data. Man I wish I would have that much SSD storage.
can you post a screenshot of your pools so I have an idea about how much free space you have on them and how big they are?
Keep in mind you are potetially making tens of snapshots per day (depending on your snaphot schedules).
They are not automatically mounted. You have to go to a specific snapshots and mount and then share that mounted snaphot.
I'll show you how tomorrow if you are interested. Now it's time for me to turn of my brain.
Let me know if you would like me to do that.
asuming your hd's are kind of modern and can maintain about 50 MB per second (thats conservatively) you should be able to replicate in about 9*1024*1024 / 50 MB/s = 188 743 seconds or 52 hours. My disks do about 160 MB/s so this could be a lot less..... but it's better to keep this in mind when you schedule your first snapshot.
9TB is a LOT of data. Man I wish I would have that much SSD storage.
can you post a screenshot of your pools so I have an idea about how much free space you have on them and how big they are?
It goes without saying (I hope) that this is only valid for the first replication. The next replication would take zero time for the snapshot and only the time to send the changed data over. This will be a fraction....
so compared with rsync you win the time to scan and compare 9 TB of data every time you want to make a new "backup".
so compared with rsync you win the time to scan and compare 9 TB of data every time you want to make a new "backup".
Actually now im curious. Why do you have ssd's for that kind of data? I would asume you have a 1 Gb network. so you can expect a max transfer rate of about 115 MB/s. That would be your bottle neck. so why do you have the ssd pool at all for this data? or do you have some extreme 40 Gb/s network?
replication should take a similar time as rsinc to write the data.... your spinning rust pool speed determines the time.
Actually i do video editing on the side and I run my Plex Media from that. I am running an Intel X540T2 10gig nic in load balancing mode for the nas and i have an ASUS xg-c100c 10gig nic in my windows tower and another Intel X540T2 10 gig nic in my Linux tower. To the SSD's I can get 1gig from my tower some times but usually it hovers around 600 to 800 MB transfers from my towers. I have a Buffalo BS-XP2012 12 port 10gig Switch which works awesome. My server sits in the rack in the oil room in my house, and i have cat 6 running to everything including my Ubiquiti acress ports. I'm one of those overkill people I hate waiting on file transfers and i don't like things buffering. I wish 40gig stuff wasn't so expensive but i'll be ready for when its cheep. lol
9TB is a LOT of data. Man I wish I would have that much SSD storage.
can you post a screenshot of your pools so I have an idea about how much free space you have on them and how big they are?[/QUOTE]
PlugIns (1) old 125 SSD for plug in's that way if something happens its an easy switch to the other set of drives.
Vault (8) WD red 4tb in a raidz some as old as 3 years others as new as last week. Decided if i need to get more space i'll just add a disk array shelf to this and add 8 disks at a time.
Zeus (12) Micron 1100 2 TB 2.5" SATA 6Gb/s SSD's in a raidz
can you post a screenshot of your pools so I have an idea about how much free space you have on them and how big they are?[/QUOTE]
PlugIns (1) old 125 SSD for plug in's that way if something happens its an easy switch to the other set of drives.
Vault (8) WD red 4tb in a raidz some as old as 3 years others as new as last week. Decided if i need to get more space i'll just add a disk array shelf to this and add 8 disks at a time.
Zeus (12) Micron 1100 2 TB 2.5" SATA 6Gb/s SSD's in a raidz
I followed your instructions to get a replication task working but it doesn't replicate. I have a test dataset pool/backuptest that I want to replicate to pool2. The backuptest dataset has been snapshot 2 times with periodic snapshot task already and once manually also.
How can I troubleshoot this?
