Snapshot status out of whack after upgrade ~20150902

[depasseg]

I opened Bug #11647 "Snapshot status out of whack after upgrade ~20150902"

https://forums.freenas.org/index.php?threads/freenas-wont-delete-old-snapshot.38174/

I have snapshots older than their expiration that are repeatedly trying to be destroyed and fail. I think this is due to the fact that there is a hold on those snapshots. I'm guessing something happened over the course of the replication related issues this month that have caused things to get out of whack. Also at least one of the snapshots trying to be destroyed doesn't even exist (tank/home@auto-20150909.1420-2w). And finally, FreeNAS is trying to perform snapshot operations on non-FreeNAS snapshots (ZREP).

Side note: After the upgrade at the beginning of September, I, like others, had issues with replication (Bugs 11130,11329, and 11504) (namely the same huge list of error messages every minute as well as the incorrect "Last snapshot sent to remote side"). I'm wondering if it's actually a snapshot issue.

Show : Logs and details

Code:

Sep 23 14:16:03 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs list -H -t snapshot -p -o name,creation -r 'backup-tank/stuff'"
Sep 23 14:16:44 freenas1 manage.py: [common.pipesubr:71] Popen()ing: klist
Sep 23 14:17:02 freenas1 autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "tank/home@auto-20150908.1500-2w"
Sep 23 14:17:03 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs list -H -t snapshot -p -o name,creation -r 'backup-tank/stuff'"
Sep 23 14:18:02 freenas1 autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "tank/home@auto-20150908.1500-2w"
Sep 23 14:18:03 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs list -H -t snapshot -p -o name,creation -r 'backup-tank/stuff'"
Sep 23 14:19:02 freenas1 autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "tank/home@auto-20150908.1500-2w"
Sep 23 14:19:03 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs list -H -t snapshot -p -o name,creation -r 'backup-tank/stuff'"
Sep 23 14:20:02 freenas1 autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs snapshot -r "tank/home@auto-20150923.1420-2w"
Sep 23 14:20:03 freenas1 autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "tank/home@auto-20150908.1500-2w"
Sep 23 14:20:03 freenas1 autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "tank/home@auto-20150909.1420-2w"
Sep 23 14:20:04 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs list -H -t snapshot -p -o name,creation -r 'backup-tank/stuff'"

And in this specific case, the snapshot that is trying to be destroyed doesn't exist:

Sep 23 14:20:03 freenas1 autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "tank/home@auto-20150909.1420-2w"

Here is the snapshot and hold status for tank/home (as one example):

[root@freenas1] ~# zfs list -r -t snapshot -o name,creation tank/home
NAME CREATION
tank/home@auto-20150908.1500-2w Tue Sep 8 15:00 2015
tank/home@auto-20150909.1720-2w Wed Sep 9 17:20 2015
tank/home@auto-20150909.2020-2w Wed Sep 9 20:20 2015
tank/home@auto-20150909.2320-2w Wed Sep 9 23:20 2015
tank/home@auto-20150910.0220-2w Thu Sep 10 2:20 2015
tank/home@auto-20150910.0520-2w Thu Sep 10 5:20 2015
tank/home@auto-20150910.0820-2w Thu Sep 10 8:20 2015
tank/home@auto-20150910.1120-2w Thu Sep 10 11:20 2015
tank/home@auto-20150910.1420-2w Thu Sep 10 14:20 2015
tank/home@auto-20150910.1720-2w Thu Sep 10 17:20 2015
tank/home@auto-20150910.2020-2w Thu Sep 10 20:20 2015
tank/home@auto-20150910.2320-2w Thu Sep 10 23:20 2015
tank/home@auto-20150911.0220-2w Fri Sep 11 2:20 2015
tank/home@auto-20150911.0520-2w Fri Sep 11 5:20 2015
tank/home@auto-20150911.0820-2w Fri Sep 11 8:20 2015
tank/home@auto-20150911.1120-2w Fri Sep 11 11:20 2015
tank/home@auto-20150911.1420-2w Fri Sep 11 14:20 2015
tank/home@auto-20150911.1720-2w Fri Sep 11 17:20 2015
tank/home@auto-20150911.2020-2w Fri Sep 11 20:20 2015
tank/home@auto-20150911.2320-2w Fri Sep 11 23:20 2015
tank/home@auto-20150912.0220-2w Sat Sep 12 2:20 2015
tank/home@auto-20150912.0520-2w Sat Sep 12 5:20 2015
tank/home@auto-20150912.0820-2w Sat Sep 12 8:20 2015
tank/home@auto-20150912.1120-2w Sat Sep 12 11:20 2015
tank/home@auto-20150912.1420-2w Sat Sep 12 14:20 2015
tank/home@auto-20150912.1720-2w Sat Sep 12 17:20 2015
tank/home@auto-20150912.2020-2w Sat Sep 12 20:20 2015
tank/home@auto-20150912.2320-2w Sat Sep 12 23:20 2015
tank/home@auto-20150913.0220-2w Sun Sep 13 2:20 2015
tank/home@auto-20150913.0520-2w Sun Sep 13 5:20 2015
tank/home@auto-20150913.0820-2w Sun Sep 13 8:20 2015
tank/home@auto-20150913.1120-2w Sun Sep 13 11:20 2015
tank/home@auto-20150913.1420-2w Sun Sep 13 14:20 2015
tank/home@auto-20150913.1720-2w Sun Sep 13 17:20 2015
tank/home@auto-20150913.2020-2w Sun Sep 13 20:20 2015
tank/home@auto-20150913.2320-2w Sun Sep 13 23:20 2015
tank/home@auto-20150914.0220-2w Mon Sep 14 2:20 2015
tank/home@auto-20150914.0520-2w Mon Sep 14 5:20 2015
tank/home@auto-20150914.0820-2w Mon Sep 14 8:20 2015
tank/home@auto-20150914.1120-2w Mon Sep 14 11:20 2015
tank/home@auto-20150914.1420-2w Mon Sep 14 14:20 2015
tank/home@auto-20150914.1720-2w Mon Sep 14 17:20 2015
tank/home@auto-20150914.2020-2w Mon Sep 14 20:20 2015
tank/home@auto-20150914.2320-2w Mon Sep 14 23:20 2015
tank/home@auto-20150915.0220-2w Tue Sep 15 2:20 2015
tank/home@auto-20150915.0520-2w Tue Sep 15 5:20 2015
tank/home@auto-20150915.0820-2w Tue Sep 15 8:20 2015
tank/home@auto-20150915.1120-2w Tue Sep 15 11:20 2015
tank/home@auto-20150915.1420-2w Tue Sep 15 14:20 2015
tank/home@auto-20150915.1720-2w Tue Sep 15 17:20 2015
tank/home@auto-20150915.2020-2w Tue Sep 15 20:20 2015
tank/home@auto-20150915.2320-2w Tue Sep 15 23:20 2015
tank/home@auto-20150916.0220-2w Wed Sep 16 2:20 2015
tank/home@auto-20150916.0520-2w Wed Sep 16 5:20 2015
tank/home@auto-20150916.0820-2w Wed Sep 16 8:20 2015
tank/home@auto-20150916.1120-2w Wed Sep 16 11:20 2015
tank/home@auto-20150916.1420-2w Wed Sep 16 14:20 2015
tank/home@auto-20150916.1720-2w Wed Sep 16 17:20 2015
tank/home@auto-20150916.2020-2w Wed Sep 16 20:20 2015
tank/home@auto-20150916.2320-2w Wed Sep 16 23:20 2015
tank/home@auto-20150917.0220-2w Thu Sep 17 2:20 2015
tank/home@auto-20150917.0520-2w Thu Sep 17 5:20 2015
tank/home@auto-20150917.0820-2w Thu Sep 17 8:20 2015
tank/home@auto-20150917.1120-2w Thu Sep 17 11:20 2015
tank/home@auto-20150917.1420-2w Thu Sep 17 14:20 2015
tank/home@auto-20150917.1720-2w Thu Sep 17 17:20 2015
tank/home@auto-20150917.2020-2w Thu Sep 17 20:20 2015
tank/home@auto-20150917.2320-2w Thu Sep 17 23:20 2015
tank/home@auto-20150918.0220-2w Fri Sep 18 2:20 2015
tank/home@auto-20150918.0520-2w Fri Sep 18 5:20 2015
tank/home@auto-20150918.0820-2w Fri Sep 18 8:20 2015
tank/home@auto-20150918.1120-2w Fri Sep 18 11:20 2015
tank/home@auto-20150918.1420-2w Fri Sep 18 14:20 2015
tank/home@auto-20150918.1720-2w Fri Sep 18 17:20 2015
tank/home@auto-20150918.2020-2w Fri Sep 18 20:20 2015
tank/home@auto-20150918.2320-2w Fri Sep 18 23:20 2015
tank/home@auto-20150919.0220-2w Sat Sep 19 2:20 2015
tank/home@auto-20150919.0520-2w Sat Sep 19 5:20 2015
tank/home@auto-20150919.0820-2w Sat Sep 19 8:20 2015
tank/home@auto-20150919.1120-2w Sat Sep 19 11:20 2015
tank/home@auto-20150919.1420-2w Sat Sep 19 14:20 2015
tank/home@auto-20150919.1720-2w Sat Sep 19 17:20 2015
tank/home@auto-20150919.2020-2w Sat Sep 19 20:20 2015
tank/home@auto-20150919.2320-2w Sat Sep 19 23:20 2015
tank/home@auto-20150920.0220-2w Sun Sep 20 2:20 2015
tank/home@auto-20150920.0520-2w Sun Sep 20 5:20 2015
tank/home@auto-20150920.0820-2w Sun Sep 20 8:20 2015
tank/home@auto-20150920.1120-2w Sun Sep 20 11:20 2015
tank/home@auto-20150920.1420-2w Sun Sep 20 14:20 2015
tank/home@auto-20150920.1720-2w Sun Sep 20 17:20 2015
tank/home@auto-20150920.2020-2w Sun Sep 20 20:20 2015
tank/home@auto-20150920.2320-2w Sun Sep 20 23:20 2015
tank/home@auto-20150921.0220-2w Mon Sep 21 2:20 2015
tank/home@auto-20150921.0520-2w Mon Sep 21 5:20 2015
tank/home@auto-20150921.0820-2w Mon Sep 21 8:20 2015
tank/home@auto-20150921.1120-2w Mon Sep 21 11:20 2015
tank/home@auto-20150921.1420-2w Mon Sep 21 14:20 2015
tank/home@auto-20150921.1720-2w Mon Sep 21 17:20 2015
tank/home@auto-20150921.2020-2w Mon Sep 21 20:20 2015
tank/home@auto-20150921.2320-2w Mon Sep 21 23:20 2015
tank/home@auto-20150922.0220-2w Tue Sep 22 2:20 2015
tank/home@auto-20150922.0520-2w Tue Sep 22 5:20 2015
tank/home@auto-20150922.0820-2w Tue Sep 22 8:20 2015
tank/home@auto-20150922.1120-2w Tue Sep 22 11:20 2015
tank/home@zrep_000130 Tue Sep 22 12:00 2015
tank/home@auto-20150922.1420-2w Tue Sep 22 14:20 2015
tank/home@auto-20150922.1720-2w Tue Sep 22 17:20 2015
tank/home@zrep_000131 Tue Sep 22 18:00 2015
tank/home@auto-20150922.2020-2w Tue Sep 22 20:20 2015
tank/home@auto-20150922.2320-2w Tue Sep 22 23:20 2015
tank/home@zrep_000132 Wed Sep 23 0:00 2015
tank/home@auto-20150923.0220-2w Wed Sep 23 2:20 2015
tank/home@auto-20150923.0520-2w Wed Sep 23 5:20 2015
tank/home@zrep_000133 Wed Sep 23 6:00 2015
tank/home@auto-20150923.0820-2w Wed Sep 23 8:20 2015
tank/home@auto-20150923.1120-2w Wed Sep 23 11:20 2015
tank/home@zrep_000134 Wed Sep 23 12:00 2015
tank/home@auto-20150923.1420-2w Wed Sep 23 14:20 2015
[root@freenas1] ~# zfs holds tank/home@auto-20150908.1500-2w
NAME TAG TIMESTAMP
tank/home@auto-20150908.1500-2w freenas:repl Tue Sep 8 15:00 2015
[root@freenas1] ~# zfs holds tank/home@auto-20150909.1420-2w
cannot open 'tank/home@auto-20150909.1420-2w': dataset does not exist
NAME TAG TIMESTAMP
no datasets available
[root@freenas1] ~#

Non-FreeNAS Snapshots trying to be destroyed:
[root@freenas1] ~# cat /var/log/messages | grep 'zrep'
Sep 23 00:00:30 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/backup@zrep_00012e'"
Sep 23 06:00:30 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/backup@zrep_00012f'"
Sep 23 10:11:57 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/btsync_1@zrep_00012e'"
Sep 23 10:11:57 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/btsync_1@zrep_00012d'"
Sep 23 10:12:01 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/crashplan_1@zrep_00012e'"
Sep 23 10:12:01 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/crashplan_1@zrep_00012d'"
Sep 23 10:12:04 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/crashplan_2@zrep_00012e'"
Sep 23 10:12:04 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/crashplan_2@zrep_00012d'"
Sep 23 10:12:08 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/htpc-manager_1@zrep_00012e'"
Sep 23 10:12:09 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/htpc-manager_1@zrep_00012d'"
Sep 23 10:12:13 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/mineos_1@zrep_00012d'"
Sep 23 10:12:13 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/mineos_1@zrep_00012c'"
Sep 23 10:12:19 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/owncloud_1@zrep_00012d'"
Sep 23 10:12:19 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/owncloud_1@zrep_00012c'"
Sep 23 10:12:54 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/plexmediaserver_1@zrep_00012d'"
Sep 23 10:12:54 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/plexmediaserver_1@zrep_00012c'"
Sep 23 10:13:09 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/sabnzbd_1@zrep_00012d'"
Sep 23 10:13:09 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/jails/sabnzbd_1@zrep_00012c'"
Sep 23 10:39:02 freenas1 notifier: cannot release hold from snapshot 'tank/backup@zrep_000130': no such tag on this dataset
Sep 23 10:39:02 freenas1 notifier: cannot release hold from snapshot 'tank/backup@zrep_000131': no such tag on this dataset
Sep 23 10:39:02 freenas1 notifier: cannot release hold from snapshot 'tank/backup@zrep_000132': no such tag on this dataset
Sep 23 10:39:02 freenas1 notifier: cannot release hold from snapshot 'tank/backup@zrep_000133': no such tag on this dataset
Sep 23 10:39:02 freenas1 notifier: cannot release hold from snapshot 'tank/backup@zrep_000134': no such tag on this dataset
Sep 23 10:42:20 freenas1 notifier: cannot release hold from snapshot 'tank/jails/btsync_1@zrep_00012f': no such tag on this dataset
Sep 23 10:42:20 freenas1 notifier: cannot release hold from snapshot 'tank/jails/btsync_1@zrep_000130': no such tag on this dataset
Sep 23 10:42:20 freenas1 notifier: cannot release hold from snapshot 'tank/jails/btsync_1@zrep_000131': no such tag on this dataset
Sep 23 10:42:20 freenas1 notifier: cannot release hold from snapshot 'tank/jails/btsync_1@zrep_000132': no such tag on this dataset
Sep 23 10:42:20 freenas1 notifier: cannot release hold from snapshot 'tank/jails/btsync_1@zrep_000133': no such tag on this dataset
Sep 23 10:42:21 freenas1 notifier: cannot release hold from snapshot 'tank/jails/crashplan_1@zrep_00012f': no such tag on this dataset
Sep 23 10:42:21 freenas1 notifier: cannot release hold from snapshot 'tank/jails/crashplan_1@zrep_000130': no such tag on this dataset
Sep 23 10:42:21 freenas1 notifier: cannot release hold from snapshot 'tank/jails/crashplan_1@zrep_000131': no such tag on this dataset
Sep 23 10:42:21 freenas1 notifier: cannot release hold from snapshot 'tank/jails/crashplan_1@zrep_000132': no such tag on this dataset
Sep 23 10:42:22 freenas1 notifier: cannot release hold from snapshot 'tank/jails/crashplan_1@zrep_000133': no such tag on this dataset
Sep 23 10:42:23 freenas1 notifier: cannot release hold from snapshot 'tank/jails/crashplan_2@zrep_00012f': no such tag on this dataset
Sep 23 10:42:23 freenas1 notifier: cannot release hold from snapshot 'tank/jails/crashplan_2@zrep_000130': no such tag on this dataset
Sep 23 10:42:23 freenas1 notifier: cannot release hold from snapshot 'tank/jails/crashplan_2@zrep_000131': no such tag on this dataset
Sep 23 10:42:23 freenas1 notifier: cannot release hold from snapshot 'tank/jails/crashplan_2@zrep_000132': no such tag on this dataset
Sep 23 10:42:23 freenas1 notifier: cannot release hold from snapshot 'tank/jails/crashplan_2@zrep_000133': no such tag on this dataset
Sep 23 10:42:24 freenas1 notifier: cannot release hold from snapshot 'tank/jails/htpc-manager_1@zrep_00012f': no such tag on this dataset
Sep 23 10:42:24 freenas1 notifier: cannot release hold from snapshot 'tank/jails/htpc-manager_1@zrep_000130': no such tag on this dataset
Sep 23 10:42:24 freenas1 notifier: cannot release hold from snapshot 'tank/jails/htpc-manager_1@zrep_000131': no such tag on this dataset
Sep 23 10:42:24 freenas1 notifier: cannot release hold from snapshot 'tank/jails/htpc-manager_1@zrep_000132': no such tag on this dataset
Sep 23 10:42:24 freenas1 notifier: cannot release hold from snapshot 'tank/jails/htpc-manager_1@zrep_000133': no such tag on this dataset
Sep 23 10:42:25 freenas1 notifier: cannot release hold from snapshot 'tank/jails/mineos_1@zrep_00012e': no such tag on this dataset
Sep 23 10:42:25 freenas1 notifier: cannot release hold from snapshot 'tank/jails/mineos_1@zrep_00012f': no such tag on this dataset
Sep 23 10:42:25 freenas1 notifier: cannot release hold from snapshot 'tank/jails/mineos_1@zrep_000130': no such tag on this dataset
Sep 23 10:42:25 freenas1 notifier: cannot release hold from snapshot 'tank/jails/mineos_1@zrep_000131': no such tag on this dataset
Sep 23 10:42:25 freenas1 notifier: cannot release hold from snapshot 'tank/jails/mineos_1@zrep_000132': no such tag on this dataset
Sep 23 10:42:26 freenas1 notifier: cannot release hold from snapshot 'tank/jails/owncloud_1@zrep_00012e': no such tag on this dataset
Sep 23 10:42:26 freenas1 notifier: cannot release hold from snapshot 'tank/jails/owncloud_1@zrep_00012f': no such tag on this dataset
Sep 23 10:42:26 freenas1 notifier: cannot release hold from snapshot 'tank/jails/owncloud_1@zrep_000130': no such tag on this dataset
Sep 23 10:42:26 freenas1 notifier: cannot release hold from snapshot 'tank/jails/owncloud_1@zrep_000131': no such tag on this dataset
Sep 23 10:42:26 freenas1 notifier: cannot release hold from snapshot 'tank/jails/owncloud_1@zrep_000132': no such tag on this dataset
Sep 23 10:42:27 freenas1 notifier: cannot release hold from snapshot 'tank/jails/plexmediaserver_1@zrep_00012e': no such tag on this dataset
Sep 23 10:42:27 freenas1 notifier: cannot release hold from snapshot 'tank/jails/plexmediaserver_1@zrep_00012f': no such tag on this dataset
Sep 23 10:42:27 freenas1 notifier: cannot release hold from snapshot 'tank/jails/plexmediaserver_1@zrep_000130': no such tag on this dataset
Sep 23 10:42:27 freenas1 notifier: cannot release hold from snapshot 'tank/jails/plexmediaserver_1@zrep_000131': no such tag on this dataset
Sep 23 10:42:27 freenas1 notifier: cannot release hold from snapshot 'tank/jails/plexmediaserver_1@zrep_000132': no such tag on this dataset
Sep 23 10:42:29 freenas1 notifier: cannot release hold from snapshot 'tank/jails/sabnzbd_1@zrep_00012e': no such tag on this dataset
Sep 23 10:42:29 freenas1 notifier: cannot release hold from snapshot 'tank/jails/sabnzbd_1@zrep_00012f': no such tag on this dataset
Sep 23 10:42:29 freenas1 notifier: cannot release hold from snapshot 'tank/jails/sabnzbd_1@zrep_000130': no such tag on this dataset
Sep 23 10:42:29 freenas1 notifier: cannot release hold from snapshot 'tank/jails/sabnzbd_1@zrep_000131': no such tag on this dataset
Sep 23 10:42:29 freenas1 notifier: cannot release hold from snapshot 'tank/jails/sabnzbd_1@zrep_000132': no such tag on this dataset
Sep 23 12:14:42 freenas1 notifier: cannot release hold from snapshot 'tank/media@zrep_00012f': no such tag on this dataset
Sep 23 12:14:42 freenas1 notifier: cannot release hold from snapshot 'tank/media@zrep_000130': no such tag on this dataset
Sep 23 12:14:42 freenas1 notifier: cannot release hold from snapshot 'tank/media@zrep_000131': no such tag on this dataset
Sep 23 12:14:42 freenas1 notifier: cannot release hold from snapshot 'tank/media@zrep_000132': no such tag on this dataset
Sep 23 12:14:42 freenas1 notifier: cannot release hold from snapshot 'tank/media@zrep_000133': no such tag on this dataset
Sep 23 12:17:33 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/stuff@zrep_00012d'"
Sep 23 12:17:34 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/stuff@zrep_00012e'"
Sep 23 12:17:34 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 127.0.0.1 "zfs destroy -d 'backup-tank/stuff@zrep_00012c'"
[root@freenas1] ~#

 

[rogerh]

On updating to 201509160044, which I think is the first of the 9.3.1 updates that actually does snapshots and replication properly, I had to delete snapshots made under 20150629 or before manually. First of all by removing the hold, which in most cases destroyed them as they were subject to 'deferred destruction' but some had to be destroyed manually. (If you are replicating them then even more needed to be destroyed manually on the destination server, as the new replication task is a bit more fastidious in this respect.) Having done this, everything worked properly[1] thereafter, though I am not sure that I haven't got some 1 year life snapshots that may need manually fixing next year!

[1] Except for the 'last sent' column and some error messages from the destination server, which haven't been fixed yet.

 

[depasseg]

I'm sure I can log into the CLI and release the holds, but the bigger issue is that the system didn't handle the issue properly.

 

[rogerh]

I'm sure I can log into the CLI and release the holds, but the bigger issue is that the system didn't handle the issue properly. :)

Fair enough, but I believe it is purely a transitional issue from the old system of replication. I don't think the new one uses holds at all. There was supposed to be a transitional system to account for this, but I guess (I don't know) that the various broken bits in the first version of the new replication stopped it working properly. For instance, I don't know if someone upgrading from 9.3 to 9.3.1 now would have a better experience of the transition. I don't fancy doing the experiment though!

 

[Redcoat]

Guys, trying to sort out any messes I made after misdiagnosing the situation when replication went to hell. I am assuming that there is a better way to delete a large number of snapshots through the CLI rather than a few at a time through the GUI. Can you advise me how, please?

 

[depasseg]

You can select all in the GUI. I filter on the dataset I want to work on, then click the view all (or 100), and then use shift-click on the top and bottom snapshots in the list, then click delete.

I'm sure there is a CLI way too.

What I found is that if there is a hold on one of the snapshots in the GUI method, the delete fails on that snapshot and then the job stops. It doesn't attempt to delete the remaining selected snapshots.

 

[Redcoat]

depasseg, many thanks, I have it now.

I thought it must have failed earlier because I had some/one with a hold as you explained, but I just selected a set using what I thought were the criteria I used earlier (?) and they deleted just fine.

I think perhaps I may have not waited long enough to get the "are you sure" or whatever prompt and just assumed the process had quit.
Thank you again.

 

[depasseg]

I'm happy to hear you've got it worked out now.