ZFS dataset / snapshots size math

Brad T · Nov 23, 2016

Hi gurus,

Hopefully a quick question. I'm using a couple of FreeNAS servers to host TimeMachine backups, mostly successfully.

I limit my "vol01/TimeMachine" datasets to 4T, which causes TimeMachine to roll off backups at the rate I want.

To improve the performance, I've

paused the TM runs
removed the 4T dataset size limit so I can perform the next step
upsized the band files in the .sparsebundle/bands directory from 8 MB to 128 MB (Established practice using OSX hdituil to a temp.sparsebundle directory)
I've checked the new "big band" sparse bundle and it checks out ok
moved the old .sparsebundle directory to a different dataset
renamed temp.sparsebundle to the real name
recreated the AFP database using the dbd command.
[next step] Want to reset the dataset size limit to 4T and resume TimeMachine service

The GUI says I am using 4.6T on the dataset, so obviously I can't set the limit to be 4T.

Being curious, I wanted to see what the 4.6T is...

From the command line (df or du), the filesystem looks to be 2.4T used. Looking at the ZFS stats, I see that 2.29T is assigned to the two snapshots that I've held on to as well. So that adds up to 4.6T quite nicely.

Code:

freenas01# df -kh /mnt/vol01/TimeMachine/
Filesystem		   Size	Used   Avail Capacity  Mounted on
vol01/TimeMachine	2.4T	2.3T	101G	96%	/mnt/vol01/TimeMachine

freenas01# du -kshc /mnt/vol01/TimeMachine/*
584G	/mnt/vol01/TimeMachine/Brad's MBA 2012.sparsebundle
1.7T	/mnt/vol01/TimeMachine/Brad's MBP 2011.sparsebundle
22M	/mnt/vol01/TimeMachine/dbd_rebuild_TimeMachine_2016-09-26_01.log
97K	/mnt/vol01/TimeMachine/dbd_rebuild_TimeMachine_2016-11-22_01.log
1.1M	/mnt/vol01/TimeMachine/dbd_rebuild_TimeMachine_2016-11-22_02.log
2.3T	total

freenas01# zfs list -r -o space vol01/TimeMachine
NAME			   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
vol01/TimeMachine   101G  4.60T	 2.29T   2.31T			  0		  0

freenas01# zfs list -t snapshot -r -o space vol01/TimeMachine
NAME					AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
vol01/TimeMachine@auto-20161009.1214-1w	  -   436M		 -	   -			  -		  -
vol01/TimeMachine@auto-20161012.0014-1w	  -  1.00G		 -	   -			  -		  -

What I'm confused about is that the snapshots "only" seem to add up to ~ 1.4G from the last command.

Can anyone tell me how i'm misinterpreting the numbers or what I'm failing to take into account to make the snapshots add up to the 2.31T that is reported in the USEDSNAP attribute from the second last command?

thanks in advance... Brad :)

darkwarrior · Nov 24, 2016

hi there,

What is your output when you are running the following command?

Code:

zfs get all vol01/TimeMachine

Should give you the values of "used","logicalused","referenced","usedbysnapshot","usedbychildren".
Maybe that will shed some light on what is consuming your disk space ;-)

BTW: df and du are practically useless on ZFS

Brad T · Nov 24, 2016

Thanks for your reply @darkwarrior.

I'm definitely seeing that du & df have limits on ZFS. ;)

Here's the output from zfs get all...

Code:

freenas01# zfs get all vol01/TimeMachine

NAME			   PROPERTY			  VALUE				   SOURCE
vol01/TimeMachine  type				  filesystem			  -
vol01/TimeMachine  creation			  Wed May  6 23:54 2015   -
vol01/TimeMachine  used				  4.60T				   -
vol01/TimeMachine  available			 101G					-
vol01/TimeMachine  referenced			2.31T				   -
vol01/TimeMachine  compressratio		 1.06x				   -
vol01/TimeMachine  mounted			   yes					 -
vol01/TimeMachine  quota				 4.70T				   local
vol01/TimeMachine  reservation		   none					local
vol01/TimeMachine  recordsize			128K					default
vol01/TimeMachine  mountpoint			/mnt/vol01/TimeMachine  default
vol01/TimeMachine  sharenfs			  off					 default
vol01/TimeMachine  checksum			  on					  default
vol01/TimeMachine  compression		   lz4					 inherited from vol01
vol01/TimeMachine  atime				 off					 inherited from vol01
vol01/TimeMachine  devices			   on					  default
vol01/TimeMachine  exec				  on					  default
vol01/TimeMachine  setuid				on					  default
vol01/TimeMachine  readonly			  off					 default
vol01/TimeMachine  jailed				off					 default
vol01/TimeMachine  snapdir			   hidden				  default
vol01/TimeMachine  aclmode			   passthrough			 local
vol01/TimeMachine  aclinherit			passthrough			 inherited from vol01
vol01/TimeMachine  canmount			  on					  default
vol01/TimeMachine  xattr				 off					 temporary
vol01/TimeMachine  copies				1					   default
vol01/TimeMachine  version			   5					   -
vol01/TimeMachine  utf8only			  off					 -
vol01/TimeMachine  normalization		 none					-
vol01/TimeMachine  casesensitivity	   sensitive			   -
vol01/TimeMachine  vscan				 off					 default
vol01/TimeMachine  nbmand				off					 default
vol01/TimeMachine  sharesmb			  off					 default
vol01/TimeMachine  refquota			  none					local
vol01/TimeMachine  refreservation		none					local
vol01/TimeMachine  primarycache		  all					 default
vol01/TimeMachine  secondarycache		all					 default
vol01/TimeMachine  usedbysnapshots	   2.29T				   -
vol01/TimeMachine  usedbydataset		 2.31T				   -
vol01/TimeMachine  usedbychildren		0					   -
vol01/TimeMachine  usedbyrefreservation  0					   -
vol01/TimeMachine  logbias			   latency				 default
vol01/TimeMachine  dedup				 off					 default
vol01/TimeMachine  mlslabel									  -
vol01/TimeMachine  sync				  standard				default
vol01/TimeMachine  refcompressratio	  1.06x				   -
vol01/TimeMachine  written			   2.31T				   -
vol01/TimeMachine  logicalused		   4.91T				   -
vol01/TimeMachine  logicalreferenced	 2.46T				   -
vol01/TimeMachine  volmode			   default				 default
vol01/TimeMachine  filesystem_limit	  none					default
vol01/TimeMachine  snapshot_limit		none					default
vol01/TimeMachine  filesystem_count	  none					default
vol01/TimeMachine  snapshot_count		none					default
vol01/TimeMachine  redundant_metadata	all					 default

darkwarrior · Nov 24, 2016

Brad T said:

Thanks for your reply @darkwarrior.

I'm definitely seeing that du & df have limits on ZFS. ;)

Here's the output from zfs get all...

Code:

freenas01# zfs get all vol01/TimeMachine

NAME			   PROPERTY			  VALUE				   SOURCE
vol01/TimeMachine  type				  filesystem			  -
vol01/TimeMachine  creation			  Wed May  6 23:54 2015   -
vol01/TimeMachine  used				  4.60T				   -
vol01/TimeMachine  available			 101G					-
vol01/TimeMachine  referenced			2.31T				   -
vol01/TimeMachine  compressratio		 1.06x				   -
vol01/TimeMachine  mounted			   yes					 -
vol01/TimeMachine  quota				 4.70T				   local
vol01/TimeMachine  reservation		   none					local
vol01/TimeMachine  recordsize			128K					default
[... snip...]
vol01/TimeMachine  usedbysnapshots	   2.29T				   -
vol01/TimeMachine  usedbydataset		 2.31T				   -
vol01/TimeMachine  usedbychildren		0					   -
[.. snip...]				-
vol01/TimeMachine  written			   2.31T				   -
vol01/TimeMachine  logicalused		   4.91T				   -
vol01/TimeMachine  logicalreferenced	 2.46T				   -
[... snip ...]

Ah, there you have it. 2.29TiB eaten up by snapshots ;-)
You can use the command below to see all datasets and snapshots (and the space they occupy on your pool). Output might be a bit long if you have many datasets

Code:

zfs get used,logicalused,usedbysnapshots vol01

Here's the link to the documentation concerning snapshots
http://doc.freenas.org/9.10/storage.html#snapshots

Brad T · Nov 24, 2016

Cheers...

I guess my question is this:

Why doesn't the output from this:
zfs list -t snapshot -r -o space vol01/TimeMachine

(436M + 1.00G)

add up to the 2.29TiB reported by the 'usedbysnapshots':
zfs get used,logicalused,usedbysnapshots vol01/TimeMachine

Code:

freenas01# zfs list -t snapshot -r -o space vol01/TimeMachine
NAME									 AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
vol01/TimeMachine@auto-20161009.1214-1w	  -   436M		 -	   -			  -		  -
vol01/TimeMachine@auto-20161012.0014-1w	  -  1.00G		 -	   -			  -		  -
freenas01#
freenas01#
freenas01# zfs get used,logicalused,usedbysnapshots vol01/TimeMachine
NAME			   PROPERTY		 VALUE	 SOURCE
vol01/TimeMachine  used			 4.60T	 -
vol01/TimeMachine  logicalused	  4.91T	 -
vol01/TimeMachine  usedbysnapshots  2.29T	 -

At this point, i'm prepared to zfs destroy the snapshots, but it only looks like i'll recover 1,460M. Hardly worth the effort and won't drop me back below the 4.0T I would like to return to for this dataset...

So where is the rest of that 2.29T at?

thanks for any ideas / insight!
Brad

darkwarrior · Nov 24, 2016

Brad T said:
Cheers...

I guess my question is this:

Why doesn't the output from this:
zfs list -t snapshot -r -o space vol01/TimeMachine

(436M + 1.00G)

add up to the 2.29TiB reported by the 'usedbysnapshots':
zfs get used,logicalused,usedbysnapshots vol01/TimeMachine

Code:
freenas01# zfs list -t snapshot -r -o space vol01/TimeMachine NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD vol01/TimeMachine@auto-20161009.1214-1w - 436M - - - - vol01/TimeMachine@auto-20161012.0014-1w - 1.00G - - - - freenas01# freenas01# freenas01# zfs get used,logicalused,usedbysnapshots vol01/TimeMachine NAME PROPERTY VALUE SOURCE vol01/TimeMachine used 4.60T - vol01/TimeMachine logicalused 4.91T - vol01/TimeMachine usedbysnapshots 2.29T -

At this point, i'm prepared to zfs destroy the snapshots, but it only looks like i'll recover 1,460M. Hardly worth the effort and won't drop me back below the 4.0T I would like to return to for this dataset...

So where is the rest of that 2.29T at?

thanks for any ideas / insight!
Brad

Hi,
did you have a look at what the webinterface is reporting ?
FreeNAS being an appliance it should be the place we are looking at in the first place.

Maybe you want to check what is listed under the toplevel dataset.
zfs get used,logicalused,usedbysnapshots vol01

Brad T · Nov 25, 2016

Hi,

vol01 stats not all that interesting...

Code:

freenas01# zfs get used,logicalused,usedbysnapshots vol01
NAME   PROPERTY		 VALUE	 SOURCE
vol01  used			 11.2T	 -
vol01  logicalused	  12.0T	 -
vol01  usedbysnapshots  0		 -

vol01/TimeMachine is more interesting... but still the same numbers.

Code:

freenas01# zfs get -r used,logicalused,usedbysnapshots vol01/TimeMachine
freenas01# zfs get -r type,used,logicalused,usedbysnapshots vol01/TimeMachine
NAME									 PROPERTY		 VALUE	   SOURCE
vol01/TimeMachine						type			 filesystem  -
vol01/TimeMachine						used			 4.60T	   -
vol01/TimeMachine						logicalused	  4.91T	   -
vol01/TimeMachine						usedbysnapshots  2.29T	   -
vol01/TimeMachine@auto-20161009.1214-1w  type			 snapshot	-
vol01/TimeMachine@auto-20161009.1214-1w  used			 436M		-
vol01/TimeMachine@auto-20161009.1214-1w  logicalused	  0		   -
vol01/TimeMachine@auto-20161009.1214-1w  usedbysnapshots  -		   -
vol01/TimeMachine@auto-20161012.0014-1w  type			 snapshot	-
vol01/TimeMachine@auto-20161012.0014-1w  used			 1.00G	   -
vol01/TimeMachine@auto-20161012.0014-1w  logicalused	  0		   -
vol01/TimeMachine@auto-20161012.0014-1w  usedbysnapshots  -		   -

Here's what the Storage > volumes web page shows (with a 4.7T limit set for the dataset). Consistent with the numbers reported by ZFS command line:

I guess my question now comes down to "What other objects consume space on ZFS?"

thanks.... Brad :)

darkwarrior · Nov 26, 2016

Hey,
well... I'm afraid you are still not looking at the right page in the webinterface ;)

Here is where you really wanna look:

You will then be able to sort by "used" and should see the snapshot/s that are using all the space :D

scwst · Nov 26, 2016

Lucas and Jude have a whole chapter on understanding ZFS storage size reporting in FreeBSD Mastery: ZFS (https://www.michaelwlucas.com/os/fmzfs). I admit I skimmed it more than I read it, since so far I haven't needed to know; what I remember boils down to "it's complicated" and "df and du don't really work once you use snapshots". If you're interested, that might be a place to look it up.

fracai · Nov 30, 2016

The key point for size reporting is that if two snapshots reference the same data, neither will report using that space. So, even though your two snapshots only add up to 1.4G, it's possible that they are both referencing an additional 2TB that is shared between them. It's also possible that the 2TB is shared with the actual dataset. In this case, it does look like that's the case.

You can verify exactly how much the snapshots take up by running:

Code:

zfs destroy -n -v vol01/TimeMachine@%

That should show (-v) what snapshots would be destroyed and how much space would be freed, without actually destroying anything (-n).

Important Announcement for the TrueNAS Community.

ZFS dataset / snapshots size math

Brad T

Dabbler

darkwarrior

Patron

Brad T

Dabbler

darkwarrior

Patron

Brad T

Dabbler

darkwarrior

Patron

Brad T

Dabbler

Attachments

darkwarrior

Patron

scwst

Explorer

fracai

Guru

Similar threads