A few questions about snapshots and replication

[hansch]

Referring to:
https://www.ixsystems.com/documenta...ml?highlight=snapshot#periodic-snapshot-tasks

I have set up snapshots of a certain pool (containing backup data from other systems), first hourly, and now 6 times a day (that seems enough to me), with a retention period of 2 weeks. Replication from system Alpha to Beta is also in place, and functioning. I can see that some snapshots take quite some time to replicate. (Bandwidth is 50 Mbit max, and backup data is being written to the Alpha system almost continually.)

"Replicated data is not visible on the receiving system until the replication task completes."

I have noticed that a task has been completed because I can browse files using Midnight Commander on the Beta system.
But... I see only one version. I see many snapshots under Storage - Snapshots, which I think (almost sure) are the ones from the Alpha system.

"The target dataset on the receiving system is automatically created in read-only mode to protect the data. To mount or browse the data on the receiving system, create a clone of the snapshot and use the clone. Clones are created in read/write mode, making it possible to browse or mount them. See Snapshots for more information on creating clones."


"Snapshots record incremental changes in data. If the receiving system does not have at least one snapshot that can be used as a basis for the incremental changes in the snapshots from the sending system, there is no way to identify only the data that has changed. In this situation, the snapshots in the receiving system target dataset are removed so a complete initial copy of the new replicated data can be created."


Questions:
1. I have been taking snapshots for a while, with a retention period of 2 weeks. (BTW The backup software stores several versions / moments in time already.)
How do I know that "one snapshot (...) can be used as a basis for the incremental changes..."? Does the first snapshot contain the most data? What if it has been deleted because of the retention period? Should I stop snapshotting and start it again to start with a full, complete snapshot?

2. Is it wise to set up snapshots on the receiving (Beta) system too? It seems like overkill. Not much changes there apart from the snapshots it receives.

3. As you can see in the attachment, sorting the snapshots "by date, latest first" does not work correctly (11.2 U2). I wil search the forum and bug tracker if that issue has been reported yet. Has anyone else seen this too?

4. I cloned one of the snapshots as a test. It worked; it contains browsable files. Can I delete that clone without further consequences? (I think so, just to be sure.)

 

[Heracles]

Hey Hansch,

By default. snapshots are hidden. This is why you see only the "active" set of data and you do not see the snapshots. You can see them in the Storage section of the WebUI. If you see a snapshot there, you know it is present in your system. It is possible to browse the snapshots and access them from the CLI, but doing this kind of under-the-hood access is not recommended.

I am not sure if it is even possible to take snapshots of a dataset itself created by zfs replication of snapshots from another FreeNAS. But possible or not, there is no point doing it. The receiving side of ZFS replication is read-only, so you can not change anything in it by default. As such, there is nothing specific to the receiving server to snapshot.

To use clone is the proper way to access the content of a previous snapshot without affecting anything else. And Yes, once you are done with that clone, you can destroy it without any concern for the original. It is a clone of the original and any modification to that clone is kept inside that clone and will not touch the original. Once done with the clone, you discard it and should you need to go back to it, just re-clone the same original.

Hope this helps you understand a little better how your environment works,

 

[Apollo]

Referring to:

Questions:
1. I have been taking snapshots for a while, with a retention period of 2 weeks. (BTW The backup software stores several versions / moments in time already.)
How do I know that "one snapshot (...) can be used as a basis for the incremental changes..."? Does the first snapshot contain the most data? What if it has been deleted because of the retention period? Should I stop snapshotting and start it again to start with a full, complete snapshot?

You are confusing terminology and use case of a snapshot.
A snapshot in ZFS world only reference block of information that were added, deleted, moved, since the previous snapshot. It doesn't actually contains your data so to speak.
Think of it as follow:

When you create your pool, the information you will store on it will be stored at block 0. When the block is full, the data will be copied to the next block, block 1. the number of blocks will increase and each block will be assinged a number which is also incremented.
If the first snapshot taken only covers the first 10 blocks, upon replication ZFS will look at the blocks seen by the snapshot and will know the first 10 blocks need to be replicated. Only then will replication copy each block (copying the content of the block itself).
When you create snapshots, ZFS knows which block was last handled by the previous snapshot, in this case you will have for the second snapshot the range of Block 10 to block 1234, for instance. The next snapshot would point Block 1235 to Block NNN and so forth.
The size of the data pointed to a snapshot is dependent of the amount of data stored. For instance you could have hundreds of snapshot covering the last several weeks, but you didn't copy any data to the corresponding dataset. Then one day you decide to copy an insane amount of data.
Let's say you copy the data and no snapshot have been taken since you started copying it. At the end of the copy, a snapshot is taken.
Replicating that last snapshot will cause the replication to send the entire 100GB of data before it completes replication.

If you were to delete the first snapshot fro the above example, you will be destroying the snapshot itself for block 0 to block 9. The second snapshot was pointing to block 10 to 1234, but because no files where deleted, then the second snapshot will be updated to include the blocks from the first snapshopt which is no longer present. So second snapshot will point to block 0 through 1234. If you deleted files stored in those block in the original snapshot, then those blocks would be freed and the content lost.

You can run a diff to see the difference between snapshots.

2. Is it wise to set up snapshots on the receiving (Beta) system too? It seems like overkill. Not much changes there apart from the snapshots it receives.

It is pointless and will cause a change of state on the replicated volume which will cause replication from Alpha to Beta to fail afterward.

3. As you can see in the attachment, sorting the snapshots "by date, latest first" does not work correctly (11.2 U2). I wil search the forum and bug tracker if that issue has been reported yet. Has anyone else seen this too?

What doesn't work exactly?
Are you thinking because the "Used" column shows 0 that the snapshot failed? That is not the case. See my previous comments.
The snapshot just didn't record changes in the blocks already present and has nothing to add to it until you add, delete or move data around.

4. I cloned one of the snapshots as a test. It worked; it contains browsable files. Can I delete that clone without further consequences? (I think so, just to be sure.)

That's correct.

 

[hansch]

Thanks for your replies. Most of it confirms what I had been thinking, and my understanding of snapshots is growing.
AFAIK, snapshots are a kind of meta data, in which the state of blocks is translated into actual files.

Still, it seems logical to me that "information" that was on system A should first be transferred to B in order to be present in whatever form on B. So, the first snapshot that is being replicated MUST be the biggest, because it contains the most information.

Here the problem arises: which one IS the first snapshot? There you get an answer to your question:
"What doesn't work exactly?"
Did you look at the dates in the screenshot? They are NOT ordered "oldest first", but more or less randomly. That is a problem, because with hundreds of snapshots, looking up the oldest (or at least: the biggest) one is very tedious when done manually.
Also, ordering by size does not work correctly.
Remember: I am doing this to be able to see if the data is complete. And, as a matter of principle, ordering SHOULD work. It's as simple as that.

(Edit: Now I see they have added new columns to the snapshot page (date etc.) where the old version only showed columns Fullname, Used and Refer. Clearly, the new columns (and maybe older columns in the process) are not calculated correctly.)

This clearly is a bug, which exists in 11.2 U2 (the latest release), which IMHO has not been tested well enough. The first test I do shows at least two bugs: sorting snapshots does not work correctly, AND the dashboard does not show any data. See this bug:

https://redmine.ixsystems.com/issues/75702

The ordering bug has not been reported (yet) by me. I have a lot of (other) work to do, and I may or may not report it.

(Request to any developers reading: please, please test any new features in a live system containing real date before releasing it. I'd rather have less features than more features which do not work correctly.)

 

[Apollo]

Thanks for your replies. Most of it confirms what I had been thinking, and my understanding of snapshots is growing.
AFAIK, snapshots are a kind of meta data, in which the state of blocks is translated into actual files.

Still, it seems logical to me that "information" that was on system A should first be transferred to B in order to be present in whatever form on B. So, the first snapshot that is being replicated MUST be the biggest, because it contains the most information.

Not true.
You are still thinking in term of files.
When you do Recursive replication and you replicate all the snapshot that exist, each snapshot will be replicated by copying each block as per described by the snapshot. No more no less.

Here the problem arises: which one IS the first snapshot? There you get an answer to your question:

ZFS sort snapshot as they are created, not by sorting them by date. It is possible your system had its clock changed.
To know which one is the first snapshot is to list it with the following command under CLI:

zfs list -t snapshot -r your_pool/your_dataset | more

The snapshot as a date at the end of its name and is for indication purpose as well as letting Freenas know when to delete the old ones.
Sometimes, when you have several automatic snapshot creation, you could have potentially some snapshot dates slightly out of order, but ZFS doesn't care about the date. ZFS looks at the metadata within the snapshot itself to tell what is the correct series of event.

"What doesn't work exactly?"
Did you look at the dates in the screenshot? They are NOT ordered "oldest first", but more or less randomly. That is a problem, because with hundreds of snapshots, looking up the oldest (or at least: the biggest) one is very tedious when done manually.
Also, ordering by size does not work correctly.
Remember: I am doing this to be able to see if the data is complete. And, as a matter of principle, ordering SHOULD work. It's as simple as that.
[

I think it is irrelevant because even though you could sort them, what would be the benefit of it? I am under the impression you want to select the largerst snapshot and then replicatre it to recover it's content, right? Well it is wrong.
As an experiment, you can create a series of manual snapshots by giving dates out of sequence and see how they turn up when listing them.
You can also run a test replication using the "-vvn" option which will show you which snapshots are going to be replicated and if a snapshot is out of sequence, you will get a ZFS error message.

ZFS replication and ZFS snapshot is very robust. If you see a snapshot exist on sida A and B, you know the data they refer to are there.
However, having different number of snapshot between side A and B is a different story and the last snapshot on both side doesn't guarranty the contnet of each pool or dataset are the same Well, not quite. The data visible on side A will be reflected on side B but cloning previous snapshot may not be the case if you don't clone the same snapshots.

(Edit: Now I see they have added new columns to the snapshot page (date etc.) where the old version only showed columns Fullname, Used and Refer. Clearly, the new columns (and maybe older columns in the process) are not calculated correctly.)

This clearly is a bug, which exists in 11.2 U2 (the latest release), which IMHO has not been tested well enough. The first test I do shows at least two bugs: sorting snapshots does not work correctly, AND the dashboard does not show any data. See this bug:

https://redmine.ixsystems.com/issues/75702

The ordering bug has not been reported (yet) by me. I have a lot of (other) work to do, and I may or may not report it.

(Request to any developers reading: please, please test any new features in a live system containing real date before releasing it. I'd rather have less features than more features which do not work correctly.)

 

[hansch]

I should read all this many, many times, and do many experiments to understand this all.

The way you describe it, there is no way to know if I have all the data present on system A is on B too. Why the hell would I then set up replication?

"ZFS sort snapshot as they are created, not by sorting them by date. It is possible your system had its clock changed.
To know which one is the first snapshot is to list it with the following command under CLI:"

System A or system B? I think you mean system A. I showed the snapshots on B, as I am most concerned with the presence of data on B, as I know it is there on A. BTW, replication is not up to date (it is now replicating a snapshot of february 17 21:00).

Time has NOT been altered on system A or B.
From the beginning I have had NTP servers set up.
And even if the time had been altered, how would the system know which is the oldest one? Surely by time stamp? And if the time stamp is wrong, there is no way the system could know how to restore the data to the right point in time.

I presume the snapshot dates are from system A, not B.

As far as I know (not having a PHD in Computer Science):
Data = ordered information. Information can be compressed, but there is a limit. Compression can only find patterns in data and then store that pattern in a more clever way. E.g. by representing 0101010101 by 5 times 01

To me, it is simple: there is no way 4 TB of (already compressed data) to be transferred over the internet as a package of e.g. 1 kB.

If there would be, we would live in a world of fairy tales.

So....
At least a few snapshots* should be big. And if I receive the first snapshot (from A to B) on Day 1, which is for example 0 kB, it CANNOT contain data as there was no other data on system B to refer to. (Maybe I should read in on the "referred to" column that is present in the snapshot page.)

Do I always have to have the LATEST snapshot to have the information in ALL snapshots?
Surely the answer must be NO. If it werre YES, one failed replication would render all data unavailable.

(I must stress that I am not concerned with snapshots on system A. I am only concerned with the answer to the question above: how do I know the information on sytem A is there on B too (up to the date of the last synced snapshot - yes!)

I am just saying that there must be a way in which a first and one time replication of information must have been completed and written all DATA (not snapshots).

* I am having an entirely new thought. What if the directory where the snapshots reside is populated with real files after the first replication is done.

I do not know how ZFS does it, but I am sure the first replication took a long time, even if snapshots were small. I guess the replication task takes data from a different source to populate the directory with files when the first replication is carried out.
That is what I saw when using mc in that directory. Actual files.

I think I am on the right path.

That would mean that replication first creates a clone of information (FILES if you want) to the "snapshot directory", after which time the snapshots only add changes to it (on a meta level).

Going back again to the manual:
"Replicated data is not visible on the receiving system until the replication task completes."

The directory where the snapshots (on B) are stored may contain many more snapshots but these I can only see in the web gui (or with the command line ZFS tools).
But when using mc, actual DATA is present. The snapshots refer to it and alter it. When a lot of files are added before a snapshot is made on system A, the snapshot must be big because this is new data. It cannot possibly be there already, so the snapshot is big. And replication will take more time (because a big snapshot takes more bandwidth and time to be transferred over the internet).
(Yes, I know a big file being changed a little only needs a small snapshot.)

 

[Apollo]

I will have to describe it in ways you can understand, at a later time.

But you really need to read up on snapshots and what they mean in ZFS, because you still don't understand the underlying structure.
In other terms you may understand if you have done some C programming in the past are the "Listes chainees" or in english "Linked list":

https://en.wikipedia.org/wiki/Linked_list:

You may want to read on Oracle ZFS:

https://docs.oracle.com/cd/E19253-01/819-5461/6n7ht6r4d/index.html

 

[hansch]

Sure. But this is about snapshots AND replication. Replication apparently uses snapshots PLUS other data.
I am not sure you understand it all yourself.

If a person really understands something, they can explain it (nofi).

You could just have said what I say in bold here.
I guess the replication task takes data from a different source to populate the directory with files when the first replication is carried out.
But you apparently did not know.

My question was based on sound logic. And I solved it:

I guess the replication task takes data from a different source to populate the directory with files when the first replication is carried out.

You mistook my quest for the answer (as outlined above) for the standard question "what is a snapshot". I understand a bit about it, as much as I need to know now.

This I all learnt by DOING it, not by reading the manual - which I am doing, I am now at page 100 of 400 of the manual. I learn much faster by doing it. Setting up the systems. See wat they do... making mistakes.

Maybe I should have made it clear that system B was empty (no info there) when replication was first set up.
But I guess that is what most people do.

I now understand this better:
"Snapshots record incremental changes in data. If the receiving system does not have at least one snapshot that can be used as a basis for the incremental changes in the snapshots from the sending system, there is no way to identify only the data that has changed. In this situation, the snapshots in the receiving system target dataset are removed so a complete initial copy of the new replicated data can be created."

There were no snapshots to be removed in the receiving (B) system because it was emtpy. A "complete initial copy of the new replicated data [was] created. That is what I saw when using mc. The changes from then on are in the snapshots.

Can you understand now why these passages were cryptic to me first, and not any more?

 

[Apollo]

I will have to give you a break down that will explain what I meant above in ways you can experience yourself.
It is a bit of work from my part and I want to do that when I have time on my hand.
There was one or more post I have gone through this exercise already, but I can't find it for now.

 

[Redcoat]

"Snapshots record incremental changes in data. If the receiving system does not have at least one snapshot that can be used as a basis for the incremental changes in the snapshots from the sending system, there is no way to identify only the data that has changed. In this situation, the snapshots in the receiving system target dataset are removed so a complete initial copy of the new replicated data can be created."

May I ask you to share the source of that statement. I think I'd like to read it and the associated material, too.

 

[Apollo]

May I ask you to share the source of that statement. I think I'd like to read it and the associated material, too.

Well, for this one, it must be from a script that is supposed to automate replication.
Under normal condition, to perform incremental replication, you need to have the last snapshot from the backup (side B) present from the source (side A). if it is, then you can replicate using the incremental option.
If none of the snapshot on side B are nowhere to be found on side A, or the last snapshot on side be is not present on side A but earlier snapshots from side B are, then ZFS will report that snapshots are already present and will not proceed away, unless you provide the -F and/or -D option to force the replication. This is only feasible if common snapshot from both side exist.

 

[hansch]

May I ask you to share the source of that statement. I think I'd like to read it and the associated material, too.

If you follow the link from my first post and scroll a bit down (towards Replication) you will find it.
https://www.ixsystems.com/documenta...s.html?highlight=snapshot#replication-options

AHA... It is there too:
"The first time a replication runs, it must duplicate data structures from the source to the destination computer. This can take much longer to complete than subsequent replications, which only send differences in data."

... data structures. Not just one snapshot, but - I guess - the complete data set from system A at a certain point in time.

 

[Redcoat]

Thanks. I looked, but did not see the text block that you quoted, just that which you referenced above. Your earlier quoted reference appeared likely to come from a source with a more comprehensive explanation than that of the iXsystems docs.

 

[hansch]

No... It is all there :)
Read carefully....

By the way, creating the clone took a short time, removing the clone took a lot of time, maybe an hour (??) because it removed more than 8.000.000 files one by one. After that, the clone directory itself could not be removed. Only after RENAMING the (empty) directory I could remove it. Strange!

 

[Redcoat]

Thanks - you got me! I ignored the warning...

 

[hansch]

You have some useful links there... thanks :)

 

[Redcoat]

They are there for my use, also when I reply to posts (the same issues come up so often and "Search", I find, doesn't work so well for me when I want to drop in a link). And I lifted a good starter kit from @Chris Moore ...

 

[hansch]

I might as well carry on in this thread: I have a snapshot replication FAILURE.
The snapshots are made via a 50 Mbit line.

I am tracking the progress:

Sending data0/miniodata0@auto-20190220.1800-2w (23%) 22-2-2019 19:30
Sending data0/miniodata0@auto-20190220.1800-2w (30%) 19:58
Sending data0/miniodata0@auto-20190220.1800-2w (80%) 23:25
Sending data0/miniodata0@auto-20190220.2200-2w (13%) 23-2-2019 8:33

It seems to go well (but a bit slow) but there is an error in the logs.

The log mail (at 01:24 yesterday) says:
The replication failed for the local ZFS data0/miniodata0 while attempting to
apply incremental send of snapshot auto-20190220.1800-2w -> auto-20190220.2200-2w to remoteserver.mydomain.com

This one is 335 GB: data0/miniodata0@auto-20190220.2200-2w

Is it retrying? I will keep an eye on this.
335 GB can be sent in approx. 18 hours over this line. With overhead, it might take a bit less than a day.
Edit: Not really:

Sending data0/miniodata0@auto-20190220.2200-2w (20%) 11:54
Sending data0/miniodata0@auto-20190220.2200-2w (22%) 13:00
Sending data0/miniodata0@auto-20190220.2200-2w (24%) 14:03
Sending data0/miniodata0@auto-20190220.2200-2w (26%) 15:17
Sending data0/miniodata0@auto-20190220.2200-2w (34%) 18:41
Sending data0/miniodata0@auto-20190220.2200-2w (44%) 23:21

That means 2 days for a snapshot of 335 GB (YES, it is new, unique data that cannot be compressed any further).

EDIT - yes, catching up!
Sending data0/miniodata0@auto-20190225.1800-2w (99%) (25-2-2019, 20:43)

 

[hansch]

A new challenge...

I was told that snapshots do not use much space.

Original system:
HEALTHY: 16.01 TiB (40%) Used / 23.93 TiB Free

Replication system:
HEALTHY: 32.17 TiB (98%) Used / 591.56 GiB Free

HUH? It is using exactly twice the amount of space for snapshots as for original data.

These are snapshots with 4 weeks retention.

How can I reduce the amount of space being used?

I have thought about reducing the period to two weeks, but it says "this does not affect snapshots that have been replicated to other systems".
So I have to (manually?) delete the old snapshots on the replication system.

So... I have to do at least 2 things:
1. Reduce the period to two weeks
2. Delete snapshots older than 2 weeks (on both systems I guess).

What happens if I delete all snapshots older than two weeks?

Will I get more space on the replication system?
Will I still have a full data set?

 

[Apollo]

A new challenge...

I was told that snapshots do not use much space.

Original system:
HEALTHY: 16.01 TiB (40%) Used / 23.93 TiB Free

Replication system:
HEALTHY: 32.17 TiB (98%) Used / 591.56 GiB Free

HUH? It is using exactly twice the amount of space for snapshots as for original data.

These are snapshots with 4 weeks retention.

How can I reduce the amount of space being used?

I have thought about reducing the period to two weeks, but it says "this does not affect snapshots that have been replicated to other systems".
So I have to (manually?) delete the old snapshots on the replication system.

So... I have to do at least 2 things:
1. Reduce the period to two weeks
2. Delete snapshots older than 2 weeks (on both systems I guess).

What happens if I delete all snapshots older than two weeks?

Will I get more space on the replication system?
Will I still have a full data set?

Snapshot don't use much space, that is correct. At least if you consider the following, for instance you copy a large file like 10GB, the data it occupies will be roughly 10GB. When you take a snapshot, the snapshot will not contain the file and take up 10GB or so. No, the snapshot will only contain the address of all the blocks pointing to the file. It could be a few 100 bytes or a few Kbyte.

When you replicate a snapshot, the "zfs send" will go through all the blocks defined in the snapshot and the block themselves are going to be copied over to the where "zfs receive" is pointing at.

If you have gazillions of snapshots, your volume could be filled up.

I suspect in your case you may have replicated one dataset and snapshots to 2 different locations. Both location containing the same data, unless you started replication but toward the end never finished and you might end up with a resumable replication.