TrueNAS-13.0-U* - Periodic Snapshots are disappearing daily

[kegloadam]

Hi Everyone,

Since 2022 February, all the snapshots created with the below configuration but disappears on a daily basis. I have recreated the tasks with adjusting all the parameters a bit but nothing has changed. Now it NAS already up-to-date with the TrueNAS-13.0-U3.1 and it is still happening. There is no other periodic snapshot tasks could harm the snapshots. When its created I can see in the pool history but next day I cannot see even in the pool history, not the creation nor the destroy.

If you have any idea, please let me know. :)

 

[Johnny Fartpants]

I've never set an expiry of 2 years before but I know 1 year and below works. Could you try and change the retention to something like 1 week just to see if that works?

 

[kegloadam]

I've never set an expiry of 2 years before but I know 1 year and below works. Could you try and change the retention to something like 1 week just to see if that works?

I have tried but the same result. And other TrueNASes with the same versions we have, there also we have 5 YEARS retention without any issue.

 

[sretalla]

Are you running any replication tasks? (those can play with snapshot retention also).

 

[kegloadam]

Are you running any replication tasks? (those can play with snapshot retention also).

I do. But I am doing also on other TrueNAS as well and there is no such issue... Also different naming convention. But honestly I can imagine anything :D Now I have created another weekly snapshot + one manual via CLI as well (already both created, will check tomorrow whether exist or any/both removed...)

 

[kegloadam]

UPDATE: The monthly snapshot is disappeared again however the weekly and the manual (CLI) created ones still exists. I will try to disable the replication tasks and we will see tomorrow's result. Note: The retention was also changed from YEARS to MONTHS

 

[kegloadam]

UPDATE2: Okay, I have sorted it out I think... In case you have replication and periodic snapshot as well for the same dataset (regardless of the naming convention of the snapshots) the replication task will remove the locally created snapshots as well in case you have custom retention set in the replication task.

Also, if you have the replication task and a local periodic snapshot task for the same dataset ( with different naming convention ), then the system does not really care about it, latter wins.

What do you think? Is it a normal behaviour? Was it the same like 1 year ago? I do remember times when I could replicate the daily snapshots from a server to another with e.g. 3 weeks retention set AND other weekly and monthly snapshots (for the same dataset) with different naming convention and retention I had...

 

[sretalla]

There's a thread somewhere where there was discussion about what process is actually removing snapshots.

What was concluded there was that zettarepl is doing it for snapshots related to replication tasks, but if I remember right, it was never really understood what's doing it (presumalbly middleware, but via what call?) for periodic snapshot tasks...

I'm not sure, but it sounds like what you're saying is that zettarepl is too greedy in destroying snapshots.

I'm not sure if it was ever recommended to have different sets of snapshots going on the one dataset, but in my experience, it was always problematic to do so and I avoid it on purpose.

 

[kegloadam]

There's a thread somewhere where there was discussion about what process is actually removing snapshots.

What was concluded there was that zettarepl is doing it for snapshots related to replication tasks, but if I remember right, it was never really understood what's doing it (presumalbly middleware, but via what call?) for periodic snapshot tasks...

I'm not sure, but it sounds like what you're saying is that zettarepl is too greedy in destroying snapshots.

I'm not sure if it was ever recommended to have different sets of snapshots going on the one dataset, but in my experience, it was always problematic to do so and I avoid it on purpose.

Thank you @sretella,

Sure I see "some" logic on what you mention however it would be good to have what I need (as was as I remember from past year...) but of course I can solve it with this procedure as well just it hurts :)

+1 I cannot really understand why this destroys are not in the zpool history... but it should be another topic :D

 

[kegloadam]

Thank you @sretella,

Sure I see "some" logic on what you mention however it would be good to have what I need (as was as I remember from past year...) but of course I can solve it with this procedure as well just it hurts :)

+1 I cannot really understand why this destroys are not in the zpool history... but it should be another topic :D

+ UPDATE: I do not understand why but this replication task removed all the locally created snapshots again... Any idea what's wrong here?