- Joined
- Dec 8, 2017
- Messages
- 442
I replicate my datasets to a remote server running freenas (11.1-U5). The snapshot tasks are set to NOT allow taking empty snapshots, and the lifetime is a week.
Some of my datasets aren't very active, and so even though the snapshot task runs every couple of hours, no new snapshots are taken because no new data has been written. For those datasets, when the snapshot expires and another is taken, that new snapshot ends up being the only snapshot for the dataset, and gets sent in its entirety to the remote server. So if the dataset is 500GB, and lifetime is a week, then once per week there is a 500GB transfer to the remote server.
Obviously this is less than ideal to have such an enormous amount of data sent when no new data has been written.
What I'm guessing is happening is that the snapshot task is generating a new snapshot, and removing the old one because it has expired, and then when the replication task runs, there are no previous snapshots to run a comparison against, and so it can't generate an incremental stream to replicate. Therefore it just starts the whole thing over again by replicating the snapshot it does have, which includes the entire dataset.
I assume that if I check "Allow taking empty snapshots" then that would fix this problem as there would be a constant stream of snapshots for the replication task to compare against. However, I wonder if there is another way to address this.
Any advice or ideas?
*EDIT* - I just realized that I can UN-check "Replicate from scratch if incremental is not possible" and that might prevent the large transfers. However, I think I had that checked originally on the replication task, and it was causing errors about the replication not being possible. Even without an error, I think it would just silently fail to replicate any data at all after the last snapshot expired.
Some of my datasets aren't very active, and so even though the snapshot task runs every couple of hours, no new snapshots are taken because no new data has been written. For those datasets, when the snapshot expires and another is taken, that new snapshot ends up being the only snapshot for the dataset, and gets sent in its entirety to the remote server. So if the dataset is 500GB, and lifetime is a week, then once per week there is a 500GB transfer to the remote server.
Obviously this is less than ideal to have such an enormous amount of data sent when no new data has been written.
What I'm guessing is happening is that the snapshot task is generating a new snapshot, and removing the old one because it has expired, and then when the replication task runs, there are no previous snapshots to run a comparison against, and so it can't generate an incremental stream to replicate. Therefore it just starts the whole thing over again by replicating the snapshot it does have, which includes the entire dataset.
I assume that if I check "Allow taking empty snapshots" then that would fix this problem as there would be a constant stream of snapshots for the replication task to compare against. However, I wonder if there is another way to address this.
Any advice or ideas?
*EDIT* - I just realized that I can UN-check "Replicate from scratch if incremental is not possible" and that might prevent the large transfers. However, I think I had that checked originally on the replication task, and it was causing errors about the replication not being possible. Even without an error, I think it would just silently fail to replicate any data at all after the last snapshot expired.
Last edited: