How to delete snapshots with dependent clones?

[sleeper52]

Version: TrueNAS 12.0-U2
I am trying to delete some old snapshots but I'm having issues with a couple of them due to dependent clones.
I followed this guide but I am stuck in trying to find the dependent clones.

Here are my pools and datasets:

Code:

root@Nasgul:/data/zfs # zpool list
NAME                SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
VMs-Plugins-Jails   928G   101G   827G        -         -    11%    10%  1.00x    ONLINE  /mnt
Volume1            10.9T  1.90T  8.98T        -         -     4%    17%  1.00x    ONLINE  /mnt
Volume2              27T  22.1T  4.90T        -         -     4%    81%  1.00x    ONLINE  /mnt
freenas-boot         59G  6.74G  52.3G        -         -     1%    11%  1.00x    ONLINE  -

This is the location of some of the snapshots I'm trying to delete:

Code:

root@Nasgul:/data/zfs # zfs list -t snapshot | grep VMs-Plugins-Jails | tail
VMs-Plugins-Jails/iocage/jails/tautulli@ioc_update_12.1-RELEASE-p11_2020-12-27_12-02-36              60K      -       96K  -
VMs-Plugins-Jails/iocage/jails/tautulli/root@ioc_update_12.1-RELEASE-p11_2020-12-27_12-02-36       1.18G      -     3.00G  -
VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@radarr                                         1.79M      -      998M  -
VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@jackett                                         680K      -      998M  -
VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@sonarr                                          696K      -      998M  -
VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@qbittorrent                                    1.85M      -      998M  -
VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@plex-plexpass                                  1.86M      -      998M  -
VMs-Plugins-Jails/iocage/releases/11.3-RELEASE/root@lidarr                                          664K      -     1.04G  -
VMs-Plugins-Jails/iocage/releases/11.3-RELEASE/root@organizr                                        664K      -     1.04G  -
VMs-Plugins-Jails/iocage/releases/11.3-RELEASE/root@tautulli

When trying to delete said snapshot, I get the following error:

Code:

root@Nasgul:/data/zfs # zfs destroy -rR "VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@jackett"
cannot unmount '/mnt/VMs-Plugins-Jails/iocage/jails/jackett/root': unmount failed
cannot destroy snapshot VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@jackett: snapshot is cloned

I tried to find the dependent clones but I get this error:

Code:

root@Nasgul:/data/zfs # zdb -d VMs-Plugins-Jails | grep "%"
zdb: can't open 'VMs-Plugins-Jails': No such file or directory

 

[Samuel Tai]

Try iocage snapremove <name of snapshot>.

 

[sleeper52]

Try iocage snapremove <name of snapshot>.

Hi Sam.

I tried but it cannot find the snapshot since it's assuming the location "VMs-Plugins-Jails/iocage/jails/".
The location of the snapshot I intend to delete is at "VMS-Plugins-Jails/iocage/releases/11.x-RELEASE/"

Code:

root@Nasgul:~ # iocage snapremove -n root@tautulli tautulli
Snapshot: VMs-Plugins-Jails/iocage/jails/tautulli@root@tautulli not found!

If I do a snaplist on the jail tautulli for instance, it shows the different snapshots that I don't intend to delete at this time. The snapshots I've been deleting are those made before 12.x-RELEASE. As you can see, the snapshots I'm trying to delete are still on 11.x-RELEASE.

Code:

root@Nasgul:/mnt/VMs-Plugins-Jails/iocage/jails/tautulli #  iocage snaplist tautulli
+------------------------------------------------------+-----------------------+-------+-------+
|                         NAME                         |        CREATED        | RSIZE | USED  |
+======================================================+=======================+=======+=======+
| ioc_update_12.1-RELEASE-p11_2020-12-27_12-02-36      | Sun Dec 27 12:02 2020 | 96K   | 60K   |
+------------------------------------------------------+-----------------------+-------+-------+
| ioc_update_12.1-RELEASE-p11_2020-12-27_12-02-36/root | Sun Dec 27 12:02 2020 | 3.00G | 1.18G |
+------------------------------------------------------+-----------------------+-------+-------+

 

[Samuel Tai]

OK, run zfs list -t snapshot -o name,clones -r VMs-Plugins-Jails/iocage. This should tell you which snapshots are the sources for which clones.

 

[sleeper52]

OK, run zfs list -t snapshot -o name,clones -r VMs-Plugins-Jails/iocage. This should tell you which snapshots are the sources for which clones.

So apparently these are the clones. I'm a bit confused because these snapshot clones are not located in the "snapshots" tab on the GUI nor are they found when using the command zfs list -t snapshot

Code:

root@Nasgul:~ # zfs list -t snapshot -o name,clones -r VMs-Plugins-Jails/iocage
NAME                                                                                               CLONES
VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@radarr                                         VMs-Plugins-Jails/iocage/jails/radarr/root
VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@jackett                                        VMs-Plugins-Jails/iocage/jails/jackett/root
VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@sonarr                                         VMs-Plugins-Jails/iocage/jails/sonarr/root
VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@qbittorrent                                    VMs-Plugins-Jails/iocage/jails/qbittorrent/root
VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@plex-plexpass                                  VMs-Plugins-Jails/iocage/jails/plex-plexpass/root
VMs-Plugins-Jails/iocage/releases/11.3-RELEASE/root@lidarr                                         VMs-Plugins-Jails/iocage/jails/lidarr/root
VMs-Plugins-Jails/iocage/releases/11.3-RELEASE/root@organizr                                       VMs-Plugins-Jails/iocage/jails/organizr/root
VMs-Plugins-Jails/iocage/releases/11.3-RELEASE/root@tautulli                                       VMs-Plugins-Jails/iocage/jails/tautulli/root

What should I do now? How do I delete these clones?

 

[Samuel Tai]

For each clone, you'll have to run zfs promote <name of clone> to break the clone linkage. Then you should be able to delete the source snapshot via zfs destroy <name of snapshot>.

 

[sleeper52]

I promoted each clone as you suggested. Somehow the original snapshot names have been changed.
Original snapshot names:

Code:

VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@radarr                                         1.79M      -      998M  -
VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@jackett                                         680K      -      998M  -
VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@sonarr                                          696K      -      998M  -
VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@qbittorrent                                    1.85M      -      998M  -
VMs-Plugins-Jails/iocage/releases/11.2-RELEASE/root@plex-plexpass                                  1.86M      -      998M  -
VMs-Plugins-Jails/iocage/releases/11.3-RELEASE/root@lidarr                                          664K      -     1.04G  -
VMs-Plugins-Jails/iocage/releases/11.3-RELEASE/root@organizr                                        664K      -     1.04G  -
VMs-Plugins-Jails/iocage/releases/11.3-RELEASE/root@tautulli

These are the names when the clones have been promoted (unlinked):

Code:

root@Nasgul:~ # zfs list -t snapshot | grep VMs-Plugins-Jails | tail
VMs-Plugins-Jails/iocage/jails/jackett/root@jackett                              1.79M      -      998M  -
VMs-Plugins-Jails/iocage/jails/plex-plexpass/root@qbittorrent                    1.85M      -      998M  -
VMs-Plugins-Jails/iocage/jails/radarr/root@radarr                                 332M      -      998M  -
VMs-Plugins-Jails/iocage/jails/sonarr/root@sonarr                                 696K      -      998M  -
VMs-Plugins-Jails/iocage/jails/tautulli/root@lidarr                               664K      -     1.04G  -
VMs-Plugins-Jails/iocage/jails/tautulli/root@organizr                             664K      -     1.04G  -

Apparently these new names also have clones:

Code:

root@Nasgul:~ # zfs list -t snapshot -o name,clones -r VMs-Plugins-Jails/iocage
NAME                                                           CLONES
VMs-Plugins-Jails/iocage/jails/jackett/root@jackett            VMs-Plugins-Jails/iocage/jails/sonarr/root
VMs-Plugins-Jails/iocage/jails/plex-plexpass/root@qbittorrent  VMs-Plugins-Jails/iocage/jails/qbittorrent/root
VMs-Plugins-Jails/iocage/jails/radarr/root@radarr              VMs-Plugins-Jails/iocage/jails/jackett/root
VMs-Plugins-Jails/iocage/jails/sonarr/root@sonarr              VMs-Plugins-Jails/iocage/jails/plex-plexpass/root
VMs-Plugins-Jails/iocage/jails/tautulli/root@lidarr            VMs-Plugins-Jails/iocage/jails/lidarr/root
VMs-Plugins-Jails/iocage/jails/tautulli/root@organizr          VMs-Plugins-Jails/iocage/jails/organizr/root

I tried promoting the clones but they keep renaming themselves to something else:

Code:

root@Nasgul:~ # zfs promote VMs-Plugins-Jails/iocage/jails/sonarr/root
root@Nasgul:~ # zfs list -t snapshot -o name,clones -r VMs-Plugins-Jails/iocage
NAME                                                           CLONES
VMs-Plugins-Jails/iocage/jails/plex-plexpass/root@qbittorrent  VMs-Plugins-Jails/iocage/jails/qbittorrent/root
VMs-Plugins-Jails/iocage/jails/radarr/root@radarr              VMs-Plugins-Jails/iocage/jails/sonarr/root
VMs-Plugins-Jails/iocage/jails/sonarr/root@jackett             VMs-Plugins-Jails/iocage/jails/jackett/root
VMs-Plugins-Jails/iocage/jails/sonarr/root@sonarr              VMs-Plugins-Jails/iocage/jails/plex-plexpass/root
VMs-Plugins-Jails/iocage/jails/tautulli/root@lidarr            VMs-Plugins-Jails/iocage/jails/lidarr/root
VMs-Plugins-Jails/iocage/jails/tautulli/root@organizr          VMs-Plugins-Jails/iocage/jails/organizr/root

Notice the VMs-Plugins-Jails/iocage/jails/sonarr/root is originally a clone of VMs-Plugins-Jails/iocage/jails/jackett/root@jackett.
After I promoted (unlinked) VMs-Plugins-Jails/iocage/jails/sonarr/root it is now a clone of VMs-Plugins-Jails/iocage/jails/radarr/root@radarr.

I also can't destroy the snapshot VMs-Plugins-Jails/iocage/jails/sonarr/root which I promoted (unlinked).

Code:

root@Nasgul:~ # zfs destroy -R VMs-Plugins-Jails/iocage/jails/sonarr/root
cannot unmount '/mnt/VMs-Plugins-Jails/iocage/jails/jackett/root': unmount failed
cannot destroy snapshot VMs-Plugins-Jails/iocage/jails/sonarr/root@jackett: snapshot is cloned
cannot destroy snapshot VMs-Plugins-Jails/iocage/jails/plex-plexpass/root@qbittorrent: snapshot is cloned
cannot destroy snapshot VMs-Plugins-Jails/iocage/jails/sonarr/root@sonarr: snapshot is cloned

I am wondering if this is some sort of bug. These snapshots were originally taken back when my plugins and jails were originally on a different RaidZ1 Pool (3x 4TB pool with dataset named "Volume1"). I transferred the plugins and jails to a different Raidz Mirror (2x 1TB SSD pool with dataset named "VMs-Plugins-Jails").

 

[make-nz]

@ sleeper52


Did you ever get rid of those snapshots? I'M in the same boat, promoting in a circle… never able to delete…

 

[cdiddy]

@sleeper52 @make-nz did either of you solve this? @Samuel Tai ? I have the same issue, i presume when a thread dies like this with no resolution it is because we were just doing something dumb, but i would love to be able to mark this "solved" and delete these snapshots

 

[sleeper52]

@sleeper52 @make-nz did either of you solve this? @Samuel Tai ? I have the same issue, i presume when a thread dies like this with no resolution it is because we were just doing something dumb, but i would love to be able to mark this "solved" and delete these snapshots

Sorry for the delayed response. No, I was never able to resolve this issue. That said, I did move a lot of the plugins to Docker. In turn, I deleted every snapshot relating to said plugins. Sorry I couldn't be of much help.

 

[compumatter]

I have run into this and solved it this way.

In my case trying to recursively delete any snapshots with 'docker' in them ie;

Code:

zfs list -t snapshot | grep -E 'docker' | awk '{ print "zfs destroy",$1}' | bash

Resulted in

• snapshot has dependent clones use '-R' to destroy the following datasets OR
• the device is busy

The device is busy turned out to be because docker was mounted by ZFS so I decided not to mess with that in case I use docker again someday.

However, thanks to ChatGPT I was able to get rid of the zfs snapshots using the following command which simply adds the flag -R to the original destroy request:

Code:

zfs list -t snapshot | grep -E 'docker' | awk '{ print "zfs destroy -R",$1}' | bash

I hope that helps someone else.