du vs zfs-list mismatch after rsync migration from jail dataset to other dataset ?

[jasonbuechler]

Updated:

I figured out most of the issue. And as predicted it's kind of a forehead slap: although I had checked my snapshots for the jail in question... I went searching for a *single* very large snapshot reflecting the sudden "deletions" last night. But I foolishly forgot that snapshots are progressive, and, of course, so were my regular archivals of security videos... thus, they nickel and dimed their way up to the "missing" space.

I'm still not fully clear on why there was no single huge snapshot covering all-at-once deletions, but I'm close enough to not care to look into it anymore. If anyone has an easily digestible explanation, please do share!

---

Ok... Sigh... I just know this is going to be a dead simple thing but after hours, I'm not even an inch closer to understanding what's happening. Please help? (And keep in mind my brain is fried with frustration at the moment!)

For months, I've been running a jail ("jail1") whose main purpose (of 2 or 3) was to ftp recordings off my IP-cameras. The recordings were saved to the jail's internal filesystem. Last night I finally got around to creating a jail for this exclusive purpose, and simultaneously move the archive to a dataset created for security recordings. I sudo'd rsync with --remove-source-files and watched the output for a while to verify things looked like they were running smoothly (i.e. that it accurately reported it was deleting original files) and went to bed with a little less than 3TB free space... and woke up with 0.2GB free space!? (And restarting the machine hasn't had an effect.)

I tried to triage the situation using the obvious tools, but different tools give me different totals, and whether or not a jail is running (aka its external mount point is mounted) will also change the totals, so I don't know what to believe, and more importantly, don't know what to do about it. Halp!

• Stock Dell PowerEdge T30
• Xeon E3-1225 v5 3.3GHz
• 8GB ram
• 4x 4TB WD Reds + 1x 8TB WD "white-red" -- RaidZ2/double redundant
• IO Crest 4 port SATA III PCI-e 2.0 x1 controller (https://www.amazon.com/dp/B00AZ9T3OU)
• Onboard ethernet

There's at least 2 confusing things going on here. I'm prepared to accept one, but then the other still makes no sense.

With the original "archival" jail ("jail1") down, iocage df reports jail1 using almost 2.91TB, but du reports that same jail using 480GB. My plex plugin should be totally unrelated to the jail1 issue, but since it's also kinda weird, I'll mention this too: At the same time, with the "plex_2" jail up, iocage df reports plex_2 using only 4.69GB, but du reports it's using 3.0TB. ...!??!
(I don't think it matters to the core problem, but both jails mount my htpc dataset. fstab is below, too.)

Code:

[fox@tree ~]$ iocage list
+-----+---------------+-------+--------------+------+
| JID |     NAME      | STATE |   RELEASE    | IP4  |
+=====+===============+=======+==============+======+
| -   | IPC_archive   | down  | 11.2-RELEASE | DHCP |
+-----+---------------+-------+--------------+------+
| -   | jail1         | down  | 11.2-RELEASE | DHCP |
+-----+---------------+-------+--------------+------+
| -   | media_sharing | down  | 11.2-RELEASE | DHCP |
+-----+---------------+-------+--------------+------+
| 2   | plex_2        | up    | 11.2-RELEASE | DHCP |
+-----+---------------+-------+--------------+------+
| 1   | transmission  | up    | 11.2-RELEASE | DHCP |
+-----+---------------+-------+--------------+------+


[fox@tree ~]$ iocage df --long
+---------------+-------+------+------+-------+------+
|     NAME      |  CRT  | RES  | QTA  |  USE  | AVA  |
+===============+=======+======+======+=======+======+
| IPC_archive   | 1.69x | none | none | 68.0M | 176G |
+---------------+-------+------+------+-------+------+
| jail1         | 1.00x | none | none | 2.91T | 176G | <===== !? most was rsync'd externally
+---------------+-------+------+------+-------+------+
| media_sharing | 2.10x | none | none | 88.8M | 176G |
+---------------+-------+------+------+-------+------+
| plex_2        | 1.63x | none | none | 4.69G | 176G | <===== iocage df shows <5G
+---------------+-------+------+------+-------+------+
| transmission  | 1.02x | none | none | 2.74G | 176G |
+---------------+-------+------+------+-------+------+

[fox@tree /tree]$ sudo du -h -d 1 /tree/iocage/jails/*
2.2T    /tree/iocage/jails/IPC_archive/root
2.2T    /tree/iocage/jails/IPC_archive
480G    /tree/iocage/jails/jail1/root
480G    /tree/iocage/jails/jail1
1.4G    /tree/iocage/jails/media_sharing/root
1.4G    /tree/iocage/jails/media_sharing
126K    /tree/iocage/jails/plex_2/plugin
3.0T    /tree/iocage/jails/plex_2/root
3.0T    /tree/iocage/jails/plex_2   <=== lies! counting external mount
118K    /tree/iocage/jails/transmission/plugin
4.0G    /tree/iocage/jails/transmission/root
4.0G    /tree/iocage/jails/transmission


[fox@tree ~/iocage/jails]$ sudo iocage fstab -l jail1
+-------+---------------------------------------------------------------------------------------------------+
| INDEX |                                            FSTAB ENTRY                                            |
+=======+===================================================================================================+
| 0     | /mnt/tree_nas/htpc      /mnt/tree_nas/iocage/jails/jail1/root/tree_htpc nullfs  rw      0       0 |
+-------+---------------------------------------------------------------------------------------------------+

[fox@tree ~/iocage/jails]$ sudo iocage fstab -l plex_2
+-------+---------------------------------------------------------------------------------------------------+
| INDEX |                                            FSTAB ENTRY                                            |
+=======+===================================================================================================+
| 14    | /mnt/tree_nas/htpc      /mnt/tree_nas/iocage/jails/plex_2/root/media    nullfs  rw      0       0 |
+-------+---------------------------------------------------------------------------------------------------+

The du utility should be smart enough to tabulate only once on links, so maybe it makes a difference running it from / ... and I accept that, but it still doesn't explain the "jail1" issue. Just for completeness, you can see du's totals change after stopping the plex plugin:

Code:

[fox@tree ~]$ sudo du -s -h /mnt/tree_nas/iocage/jails/*
Password:
1.4G    /mnt/tree_nas/iocage/jails/IPC_archive
480G    /mnt/tree_nas/iocage/jails/jail1
1.4G    /mnt/tree_nas/iocage/jails/media_sharing
3.0T    /mnt/tree_nas/iocage/jails/plex_2     <===== while jail is up: du shows 3T
4.0G    /mnt/tree_nas/iocage/jails/transmission

[fox@tree ~]$ sudo iocage stop plex_2
Password:
* Stopping plex_2
  + Running prestop OK
  + Stopping services OK
  + Tearing down VNET OK
  + Removing devfs_ruleset: 6 OK
  + Removing jail process OK
  + Running poststop OK

[fox@tree ~]$ sudo du -s -h /mnt/tree_nas/iocage/jails/*
1.4G    /mnt/tree_nas/iocage/jails/IPC_archive
480G    /mnt/tree_nas/iocage/jails/jail1
1.4G    /mnt/tree_nas/iocage/jails/media_sharing
4.3G    /mnt/tree_nas/iocage/jails/plex_2     <===== jail down: du now "almost" 0
4.0G    /mnt/tree_nas/iocage/jails/transmission

Beginning to grasp at straws, I removed the external mount from jail1, but that changed nothing in either du or iocage df. And now I'm out of ideas...

Code:

[fox@tree ~]$ sudo du -s -h /mnt/tree_nas/iocage/jails/*
1.4G    /mnt/tree_nas/iocage/jails/IPC_archive
480G    /mnt/tree_nas/iocage/jails/jail1
1.4G    /mnt/tree_nas/iocage/jails/media_sharing
4.3G    /mnt/tree_nas/iocage/jails/plex_2
4.0G    /mnt/tree_nas/iocage/jails/transmission

[fox@tree ~]$ iocage df --long
+---------------+-------+------+------+-------+------+
|     NAME      |  CRT  | RES  | QTA  |  USE  | AVA  |
+===============+=======+======+======+=======+======+
| IPC_archive   | 1.69x | none | none | 68.0M | 176G |
+---------------+-------+------+------+-------+------+
| jail1         | 1.00x | none | none | 2.91T | 176G |
+---------------+-------+------+------+-------+------+
| media_sharing | 2.10x | none | none | 88.8M | 176G |
+---------------+-------+------+------+-------+------+
| plex_2        | 1.63x | none | none | 4.69G | 176G |
+---------------+-------+------+------+-------+------+
| transmission  | 1.02x | none | none | 2.74G | 176G |
+---------------+-------+------+------+-------+------+


root@tree:/mnt/tree_nas/iocage/jails # zpool list
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freenas-boot    58G  3.87G  54.1G        -         -      -     6%  1.00x  ONLINE  -
tree_nas      18.1T  17.1T  1.02T        -         -    26%    94%  1.00x  ONLINE  /mnt

Someone please tell me I'm missing some crucial part of understanding how to read some of these outputs and there's a quick and obvious way to fix up what I'm seeing? Thaaaaanks.

--jason