SOLVED USEDCHILD data, but there is no child dataset

bluezfan1967

Cadet
Joined
Oct 16, 2020
Messages
4
I am working on setting up a backup FreeNAS server. Both the backup, and prod, servers are running FreeNAS-11.3-U4. I created a Replication task via the GUI, and was able to send the datasets I wanted replicated from the production server to the backup server. Once the initial replication process finished, I validated that the files were on the backup server, and everything seemed fine. However, then I started having errors on the production machine not sending the recent snapshots to the backup server. While troubleshooting these errors, the backup server's pool became full. I went through and deleted all the snapshots on the backup server, but this did not free up enough space to take me out of 100% utilization. I ran the command below, and see that the USEDSNAP field is empty, the USEDDS field shows the expected amount of space used by the dataset, but the USEDCHILD field is saying 2.59TB of data is being used. However, the dataset, StorageBackup/Media, does not have any children. I'm relatively new to ZFS, so is there something I'm just overlooking, or not understanding in regards to each field?

zfs list -o name,quota,refquota,reservation,refreservation,used,usedds,usedsnap,usedchild -t all StorageBackup/Media
storagebackup output.png
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,600
Please include the output of the following commands:
Code:
zfs get all StorageBackup/Media
zfs list -t all -r StorageBackup
 

bluezfan1967

Cadet
Joined
Oct 16, 2020
Messages
4
Below is the output of the commands you recommended. If I ran the command to list all of the datasets, the post met the max limit. So, I ran it with just the dataset in question, Media, and then attached the file with the list output for all child datasets of StorageBackup.

ZFS Get Output
zfs get all StorageBackup/Media
NAME PROPERTY VALUE SOURCE
StorageBackup/Media type filesystem -
StorageBackup/Media creation Mon May 11 22:24 2020 -
StorageBackup/Media used 10.1T -
StorageBackup/Media available 3.27M -
StorageBackup/Media referenced 7.50T -
StorageBackup/Media compressratio 1.00x -
StorageBackup/Media mounted yes -
StorageBackup/Media quota none default
StorageBackup/Media reservation none default
StorageBackup/Media recordsize 128K default
StorageBackup/Media mountpoint /mnt/StorageBackup/Media inherited from StorageBackup
StorageBackup/Media sharenfs off default
StorageBackup/Media checksum on default
StorageBackup/Media compression lz4 received
StorageBackup/Media atime on default
StorageBackup/Media devices on default
StorageBackup/Media exec on default
StorageBackup/Media setuid on default
StorageBackup/Media readonly off default
StorageBackup/Media jailed off default
StorageBackup/Media snapdir hidden default
StorageBackup/Media aclmode restricted received
StorageBackup/Media aclinherit passthrough inherited from StorageBackup
StorageBackup/Media createtxg 3250 -
StorageBackup/Media canmount on default
StorageBackup/Media xattr off temporary
StorageBackup/Media copies 1 received
StorageBackup/Media version 5 -
StorageBackup/Media utf8only off -
StorageBackup/Media normalization none -
StorageBackup/Media casesensitivity sensitive -
StorageBackup/Media vscan off default
StorageBackup/Media nbmand off default
StorageBackup/Media sharesmb off default
StorageBackup/Media refquota none default
StorageBackup/Media refreservation none default
StorageBackup/Media guid 1944202640104451639 -
StorageBackup/Media primarycache all default
StorageBackup/Media secondarycache all default
StorageBackup/Media usedbysnapshots 0 -
StorageBackup/Media usedbydataset 7.50T -
StorageBackup/Media usedbychildren 2.59T -
StorageBackup/Media usedbyrefreservation 0 -
StorageBackup/Media logbias latency default
StorageBackup/Media dedup off default
StorageBackup/Media mlslabel -
StorageBackup/Media sync standard default
StorageBackup/Media refcompressratio 1.00x -
StorageBackup/Media written 7.50T -
StorageBackup/Media logicalused 10.1T -
StorageBackup/Media logicalreferenced 7.55T -
StorageBackup/Media volmode default default
StorageBackup/Media filesystem_limit none default
StorageBackup/Media snapshot_limit none default
StorageBackup/Media filesystem_count none default
StorageBackup/Media snapshot_count none default
StorageBackup/Media redundant_metadata all default
StorageBackup/Media receive_resume_token 1-1113950393-f0-789c636064000310a500c4ec50360710e72765a52697303048ce81a8c1904f4b2b4e2d01c938ad6482cab321c9275596a41603e9096f26af9ac984a9bf243fbd34338581a12a2067f2993fe1131d90e439c1f27989b9a90c0cc125f94589e9a9fabea92999890e89a525f9ba46064606069646c67a86260606ba862970f7713320fc939c9f5b50945a5c9c9fcd0007002053257a -

ZFS List Output
root@Blackgate[~]# zfs list -t all -ro name,used,avail,quota,reservation,usedds,usedsnap,usedchild,mountpoint StorageBackup/Media
NAME USED AVAIL QUOTA RESERV USEDDS USEDSNAP USEDCHILD MOUNTPOINT
StorageBackup/Media 10.1T 3.27M none none 7.50T 0 2.59T /mnt/StorageBackup/Media
 

Attachments

  • list_output.txt
    54.8 KB · Views: 196

JaimieV

Guru
Joined
Oct 12, 2012
Messages
742
I'm watching with interest - I've had some broken replica transfers lately and I suspect the partials build up invisibly, as they appear to have done on yours. Looking forward to finding out how to find and flush them if that's the case.
 
Last edited:

bluezfan1967

Cadet
Joined
Oct 16, 2020
Messages
4
@JaimieV this is possible. One of the errors I was receiving after setting up the replication tasks was,
replication failed: no incremental base on dataset and replication from scratch is not allowed
Troubleshooting this error, I enabled the "Replicate from scratch if incremental is not possible" checkbox in the replication tasks. I feel not to far after enabling this option, I started getting the space issue on the backup server. I ran the zpool command below to see if anything was getting written to disk. I checked running processes to see if there was a "zombie" process for the zfs recv stuck, and am not seeing anything. If I figure out what I broke, or how to clear the partials, before anyone else here can, I'll be sure to share what I find.

zpool_iostat_output.png
 

bluezfan1967

Cadet
Joined
Oct 16, 2020
Messages
4
@JaimieV I believe I figured the issue out. You were right that the failed send jobs are being queued, and not being deleted. If you run...

zfs get receive_resume_token <YOUR DATASET>

...You may see a long string of alphanumeric characters in the Value column representing a token. To remove the token, and free up the space, I ran...

zfs recv -A <DATASET>

...From the man page...
destroy queued received state zfs.png


...I was pointed in this direction from OpenZFS Github. I was a bad admin, and unfortunately didn't verify if there was a hidden dataset with the % sign in front before running the recv -A command above. However, this is what I did to clear up my USEDCHILD issue.
 

JaimieV

Guru
Joined
Oct 12, 2012
Messages
742
Good find - your hunting went better than mine did :) I didn't know % hid datasets either, I knew . does.That is double useful knowledge, thank you very much.

(your png didn't attach btw)
 
Top