Help me understand "used"

[garm]

Okey, so usually my Dell T20 at home is a storage for home movies, pictures and documents.
For the past week i have been running ddrescue trying to salvage data off a friends collection of crumbling disks.

I'm doing this on my workstation running Ubuntu and I created a dataset for it with gzip compression and shared it with NFS from FreeNAS 11.2 - RC2. This works like a charm as the bottleneck is the broken disk being read and NFS otherwise easily saturates my 1GbE interface.

The space being used by the img files created by ddrescue is not making sens though.
Looking from Ubuntu over NFS the sizes reported are

Code:

garm@ubuntu:~$ ls -alh /nfs/rescue/
total 171G
drwxrwx--- 4 garm root   12 nov 19 21:24 .
drwxr-xr-x 3 root root 4,0K nov 18 14:06 ..
-rw-r--r-- 1 garm root 9,6G nov 19 11:06 05276483.img
-rw-r--r-- 1 garm root  354 nov 19 08:27 05276483.map
-rw-r--r-- 1 garm root 112G nov 18 17:29 071206DP1C10DJG95KVP.img
-rw-r--r-- 1 garm root  382 nov 18 17:29 071206DP1C10DJG95KVP.map
-rw-r--r-- 1 garm root  75G nov 19 05:04 3HV2YPF5.img
-rw-r--r-- 1 garm root  356 nov 18 21:23 3HV2YPF5.map
drwxr-xr-x 2 garm garm	4 nov 19 10:55 output
-rw-r--r-- 1 garm root 278G nov 20 07:02 TE95113NGHG1BS.img
-rw-r--r-- 1 garm root  45K nov 20 07:02 TE95113NGHG1BS.map

TE95113NGHG1BS.img is in the process of being written, its about 60% done.
But looking in the FreeNAS ui, the dataset has only used a total of 234.58 GiB including the 63.84 GiB in "output" (the resulting tar archive I send over when Im done). The compression ratio reported by FreeNAS is 1.20x.

This is all reported the same by zfs

Code:

[garm@nas ~]$ zfs list -o name,used,compressratio
NAME													USED  RATIO
babba/rescue											235G  1.20x
babba/rescue/output									63.8G  1.00x

What is going on? There is over 540G written to the dataset so far, but only 235G "used" with a compression ratio of 1.20?
Does the long string of zeros from the img files not count against compression?

 

[m0nkey_]

Anything that can be compressed, will be compressed if compression is enabled (lz4 by default in FreeNAS). You can look at logicalused.

Code:

[dave@tardis ~]$ zfs get used,logicalused,compressratio storage/home/dave
NAME			   PROPERTY	   VALUE  SOURCE
storage/home/dave  used		   114G   -
storage/home/dave  logicalused	176G   -
storage/home/dave  compressratio  1.55x  -
[dave@tardis ~]$

 

[danb35]

Does the long string of zeros from the img files not count against compression?

From what I've observed, no, it doesn't. To see, try running truncate -s 8T testfile in some dataset, and see what that does to the "used" and "compressratio" fields.

 

[HoneyBadger]

From what I've observed, no, it doesn't. To see, try running truncate -s 8T testfile in some dataset, and see what that does to the "used" and "compressratio" fields.

Correct.

Code:

[root@badger] /mnt/mushroom/snaake# zfs get compression,compressratio,logicalused,used mushroom/snaake

NAME			 PROPERTY	   VALUE	 SOURCE
mushroom/snaake  compression	lz4	   local
mushroom/snaake  compressratio  1.00x	 -
mushroom/snaake  logicalused	12K	   -
mushroom/snaake  used		   96K	   -

[root@badger] /mnt/mushroom/snaake# ls -al

total 1
drwxr-xr-x  2 root  wheel  2 Nov 20 13:04 ./
drwxr-xr-x  4 root  wheel  4 Nov  8 15:31 ../

[root@badger] /mnt/mushroom/snaake# dd if=/dev/zero of=testfile bs=1M count=1K

1024+0 records in
1024+0 records out
1073741824 bytes transferred in 0.608761 secs (1763815549 bytes/sec)

[root@badger] /mnt/mushroom/snaake# ls -al

total 2
drwxr-xr-x  2 root  wheel		   3 Nov 20 13:05 ./
drwxr-xr-x  4 root  wheel		   4 Nov  8 15:31 ../
-rw-r--r--  1 root  wheel  1073741824 Nov 20 13:05 testfile

[root@badger] /mnt/mushroom/snaake# zfs get compression,compressratio,logicalused,used mushroom/snaake

NAME			 PROPERTY	   VALUE	 SOURCE
mushroom/snaake  compression	lz4	   local
mushroom/snaake  compressratio  1.00x	 -
mushroom/snaake  logicalused	12K	   -
mushroom/snaake  used		   96K	   -

[root@badger] /mnt/mushroom/snaake#

 

[danb35]

Love your dataset names, @HoneyBadger. But I'm sure you don't give a sh*t. (-:

 

[garm]

I asked my then 3 year old what the dataset should be called. Without hesitation she said "babba", a turd like creature from a group of cartoons used in preschool in Sweden

Okey, I did check logicalused but that still dosent add up. There most be some other mechanism at play that wont write long stings of zero

Code:

[garm@nas ~]$ zfs get compression,compressratio,used,logicalused babba/rescue
NAME		  PROPERTY	   VALUE	 SOURCE
babba/rescue  compression	gzip-9	local
babba/rescue  compressratio  1.20x	 -
babba/rescue  used		   235G	  -
babba/rescue  logicalused	284G	  -

Anything that can be compressed, will be compressed if compression is enabled (lz4 by default in FreeNAS). You can look at logicalused.

Code:

[dave@tardis ~]$ zfs get used,logicalused,compressratio storage/home/dave
NAME			   PROPERTY	   VALUE  SOURCE
storage/home/dave  used		   114G   -
storage/home/dave  logicalused	176G   -
storage/home/dave  compressratio  1.55x  -
[dave@tardis ~]$

I have lz4 on all my datasets except those for storing databases where I use zle, on this I decided to try gzip as I want to waste as little disk as possible and dont really care about performance. ddrescue is slow anyway on this disk, in the past 9 hours is managed to write 15 GB... [offtopic] and I want to try out gzip because I have a plan of rearranging my jails so the templates and ports are stored with gzip for added efficiency. [/offtopic]

 

[garm]

From what I've observed, no, it doesn't. To see, try running truncate -s 8T testfile in some dataset, and see what that does to the "used" and "compressratio" fields.

ya thats it, zfs apparently "ignores" large sets of zeros. thats awesome!

Code:

[garm@nas /mnt/babba/trunc]$ truncate -s 100G testfile
[garm@nas /mnt/babba/trunc]$ zfs get compression,compressratio,used,logicalused babba/trunc
NAME		 PROPERTY	   VALUE	 SOURCE
babba/trunc  compression	off	   local
babba/trunc  compressratio  1.00x	 -
babba/trunc  used		   88K	   -
babba/trunc  logicalused	36.5K	 -
[garm@nas /mnt/babba/trunc]$ ls -alh
total 10
drwxr-xr-x   2 garm  garm	  3B Nov 20 17:56 .
drwxr-xr-x  11 root  wheel	11B Nov 20 17:55 ..
-rw-r--r--   1 garm  garm	100G Nov 20 17:56 testfile

 

[HoneyBadger]

Love your dataset names, @HoneyBadger. But I'm sure you don't give a sh*t. (-:

Sounds like you've found the video that spawned my username. ;)

Okey, I did check logicalused but that still dosent add up. There most be some other mechanism at play that won't write long stings of zero

Code:

[garm@nas ~]$ zfs get compression,compressratio,used,logicalused babba/rescue
NAME		  PROPERTY	   VALUE	 SOURCE
babba/rescue  compression	gzip-9	local
babba/rescue  compressratio  1.20x	 -
babba/rescue  used		   235G	  -
babba/rescue  logicalused	284G	  -

The function is called Zero Length Encoding (ZLE) and it does exactly what its name implies - it says "Hey, the following length of N bytes are all zeroes" - I believe the decision not to count zeroes in used/logicalused was intentional because a "zeroed drive" is considered 100% free.

Bonus trivia: An uncharged NAND gate actually has a value of 1. Please don't write a bunch of actual physical 0's to your SSDs to wipe them; just use your SSD's secure-erasure function. :)

 

[garm]

But what I’m not following is why ZLE is a compression option if ZFS natively already do the same? What is the difference between a dataset with ZLE and a dataset without compression (see my truncate example above)

 

[HoneyBadger]

compression=zle is literally "compress only the 0's" - the other algorithms like lzjb/lz4/etc all do that as part of their own compression, so they supersede it in that regard.

If you set compression=off then it will write actual physical 0's to the vdevs.

 

[garm]

But it didn’t, look at the example I gave. According to FreeBSD there is a 100G file in that dataset but according to ZFS the size “used” is only 88K with compression turned off?

 

[HoneyBadger]

But it didn’t, look at the example I gave. According to FreeBSD there is a 100G file in that dataset but according to ZFS the size “used” is only 88K with compression turned off?

Because truncate generates a sparse file, it isn't even attempting to send zeroes. Use something like dd if=/dev/zero of=testfile bs=1M count=1K instead and watch what happens.

Code:

[root@badger] /mnt/mushroom/snaake# zfs get compression,compressratio,logicalused,used mushroom/snaake

NAME			 PROPERTY	   VALUE	 SOURCE
mushroom/snaake  compression	off	   local
mushroom/snaake  compressratio  1.00x	 -
mushroom/snaake  logicalused	11.5K	 -
mushroom/snaake  used		   88K	   -

[root@badger] /mnt/mushroom/snaake# ls -al

total 1
drwxr-xr-x  2 root  wheel  2 Nov 20 13:06 ./
drwxr-xr-x  4 root  wheel  4 Nov  8 15:31 ../

[root@badger] /mnt/mushroom/snaake# dd if=/dev/zero of=testfile bs=1M count=1K

1024+0 records in
1024+0 records out
1073741824 bytes transferred in 0.593079 secs (1810453010 bytes/sec)

[root@badger] /mnt/mushroom/snaake# zfs get compression,compressratio,logicalused,used mushroom/snaake

NAME			 PROPERTY	   VALUE	 SOURCE
mushroom/snaake  compression	off	   local
mushroom/snaake  compressratio  1.00x	 -
mushroom/snaake  logicalused	1.00G	 -
mushroom/snaake  used		   1.00G	 -

[root@badger] /mnt/mushroom/snaake# ls -al

total 1049610
drwxr-xr-x  2 root  wheel		   3 Nov 20 13:07 ./
drwxr-xr-x  4 root  wheel		   4 Nov  8 15:31 ../
-rw-r--r--  1 root  wheel  1073741824 Nov 20 13:07 testfile

[root@badger] /mnt/mushroom/snaake#

 

[garm]

aa, I was under the impression that a sparse image will be reported as its actual size, not its potential size. But reading the ddrescue docs it states it will not write a sparse image unless expressly told to. I guess gzip is dealing with the zeros, but then I would expect "compression" to report that as well.. I'm not entirely sure what is going on.