Where did the extra space go?

[MikeyG]

I have a RAIDZ1 volume with a single vdev of 4 X 6TB drives. As expected, FreeNAS shows 21.8TB for the total size. However, the usable size is reported as 16.9TB. If my math is right, it should be about 18TB (around 3.6*5). What happened here? Is there a TB of overhead that I'm missing unique to ZFS?

Code:

root@freenas:~ # zpool list
NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
RaidZ		 21.8T  10.4T  11.4T		 -	33%	47%  1.00x  ONLINE  /mnt

root@freenas:~ # zfs list
NAME													 USED  AVAIL  REFER  MOUNTPOINT
RaidZ												   8.28T  8.56T   153K  /mnt/RaidZ

 

[Ericloewe]

If my math is right, it should be about 18TB (around 3.6*5)

It's not, you have three drives usable (one drive's worth of parity and three drives' worth of data).
3 drives x 6TB x 10^12/2^40

The number FreeNAS reports is in line with expectations.

 

[MikeyG]

Sorry, I wrote that wrong. I have 6 4TB drives, not the other way around.

 

[Ericloewe]

Okay, that does change things a bit. What's the workload here?

 

[MikeyG]

Data stored is mostly media: movies, music, etc. There's about 400GB of iSCSI in use. About 400GB time machine backup in use.

 

[Ericloewe]

By the way, you definitely should avoid calling your pool "RaidZ". That may cause you trouble in the future, so it's best to rename it.

There's about 400GB of iSCSI in use

Okay, that could account for absurd overhead on RAIDZ, but still, ~3x overhead? Okay, this is going to need some additional information, namely the output of ([CODE][/CODE] tags please - or use pastebin and link it here):
zfs list -r RaidZ
zpool list -v RaidZ
zfs get all RaidZ

 

[Ericloewe]

Also, add zfs get all RaidZ/NameOfZvol to the list, obviously replacing the placeholder path with the zvol that contains the iSCSI stuff.

 

[MikeyG]

Thanks re: the naming. I assume there's no way to change it now? Will keep that in mind for the future. Open to any other advice you have. This is serving mostly as a backup/test build while I learn the system before building something more complete in the future.

Code:

root@freenas:~ # zfs list -r RaidZ
NAME													 USED  AVAIL  REFER  MOUNTPOINT
RaidZ												   8.28T  8.56T   153K  /mnt/RaidZ
RaidZ/.system										   50.0M  8.56T   211K  legacy
RaidZ/.system/configs-d6d294a404024d4b9f621e6bf1afa53c  7.03M  8.56T  7.03M  legacy
RaidZ/.system/cores									 4.24M  8.56T  4.24M  legacy
RaidZ/.system/rrd-d6d294a404024d4b9f621e6bf1afa53c	  37.0M  8.56T  37.0M  legacy
RaidZ/.system/samba4									 486K  8.56T   486K  legacy
RaidZ/.system/syslog-d6d294a404024d4b9f621e6bf1afa53c   1.10M  8.56T  1.10M  legacy
RaidZ/Backup											7.45T  8.56T  7.03T  /mnt/RaidZ/Backup
RaidZ/Downloads										 16.2G  8.56T  16.2G  /mnt/RaidZ/Downloads
RaidZ/Downloads/Incomplete							   179K  8.56T   179K  /mnt/RaidZ/Downloads/Incomplete
RaidZ/Scratchdisk									   79.5G  8.56T  79.5G  /mnt/RaidZ/Scratchdisk
RaidZ/Storj-1										   18.5G  8.56T  18.5G  -
RaidZ/Storj-2											340G  8.56T   340G  -
RaidZ/Time Machine									   390G  59.7G   390G  /mnt/RaidZ/Time Machine
RaidZ/jails											 3.83G  8.56T   230K  /mnt/RaidZ/jails
RaidZ/jails/.warden-template-pluginjail				  608M  8.56T   607M  /mnt/RaidZ/jails/.warden-template-pluginjail
RaidZ/jails/.warden-template-standard				   2.44G  8.56T  2.44G  /mnt/RaidZ/jails/.warden-template-standard
RaidZ/jails/sonarr_1									 634M  8.56T  1.04G  /mnt/RaidZ/jails/sonarr_1
RaidZ/jails/transmission_1							   185M  8.56T   698M  /mnt/RaidZ/jails/transmission_1

Code:

root@freenas:~ # zpool list -v RaidZ
NAME									 SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
RaidZ								   21.8T  10.4T  11.4T		 -	33%	47%  1.00x  ONLINE  /mnt
  raidz1								21.8T  10.4T  11.4T		 -	33%	47%
	gptid/f05889df-aecc-11e7-ba7f-7085c23da375	  -	  -	  -		 -	  -	  -
	gptid/f0f7d82a-aecc-11e7-ba7f-7085c23da375	  -	  -	  -		 -	  -	  -
	gptid/f1ba6a88-aecc-11e7-ba7f-7085c23da375	  -	  -	  -		 -	  -	  -
	gptid/f282b084-aecc-11e7-ba7f-7085c23da375	  -	  -	  -		 -	  -	  -
	gptid/f33ccde8-aecc-11e7-ba7f-7085c23da375	  -	  -	  -		 -	  -	  -
	gptid/f3e5e9eb-aecc-11e7-ba7f-7085c23da375	  -	  -	  -		 -	  -	  -

Code:

root@freenas:~ # zfs get all RaidZ
NAME   PROPERTY				VALUE				   SOURCE
RaidZ  type					filesystem			  -
RaidZ  creation				Wed Oct 11 14:41 2017   -
RaidZ  used					8.28T				   -
RaidZ  available			   8.56T				   -
RaidZ  referenced			  153K					-
RaidZ  compressratio		   1.01x				   -
RaidZ  mounted				 yes					 -
RaidZ  quota				   none					default
RaidZ  reservation			 none					default
RaidZ  recordsize			  128K					default
RaidZ  mountpoint			  /mnt/RaidZ			  default
RaidZ  sharenfs				off					 default
RaidZ  checksum				on					  default
RaidZ  compression			 lz4					 local
RaidZ  atime				   on					  default
RaidZ  devices				 on					  default
RaidZ  exec					on					  default
RaidZ  setuid				  on					  default
RaidZ  readonly				off					 default
RaidZ  jailed				  off					 default
RaidZ  snapdir				 hidden				  default
RaidZ  aclmode				 passthrough			 local
RaidZ  aclinherit			  passthrough			 local
RaidZ  canmount				on					  default
RaidZ  xattr				   off					 temporary
RaidZ  copies				  1					   default
RaidZ  version				 5					   -
RaidZ  utf8only				off					 -
RaidZ  normalization		   none					-
RaidZ  casesensitivity		 sensitive			   -
RaidZ  vscan				   off					 default
RaidZ  nbmand				  off					 default
RaidZ  sharesmb				off					 default
RaidZ  refquota				none					default
RaidZ  refreservation		  none					default
RaidZ  primarycache			all					 default
RaidZ  secondarycache		  all					 default
RaidZ  usedbysnapshots		 0					   -
RaidZ  usedbydataset		   153K					-
RaidZ  usedbychildren		  8.28T				   -
RaidZ  usedbyrefreservation	0					   -
RaidZ  logbias				 latency				 default
RaidZ  dedup				   off					 default
RaidZ  mlslabel										-
RaidZ  sync					standard				default
RaidZ  refcompressratio		1.00x				   -
RaidZ  written				 153K					-
RaidZ  logicalused			 8.39T				   -
RaidZ  logicalreferenced	   31.5K				   -
RaidZ  volmode				 default				 default
RaidZ  filesystem_limit		none					default
RaidZ  snapshot_limit		  none					default
RaidZ  filesystem_count		none					default
RaidZ  snapshot_count		  none					default
RaidZ  redundant_metadata	  all					 default
RaidZ  org.freebsd.ioc:active  yes					 local

 

[Ericloewe]

You forgot the [CODE][/CODE] tags.

Anyway, iSCSI is on Storj-2?

 

[MikeyG]

I think I'm not understanding what you mean by " tags" or how to get the output of that. Yes, iSCSI storage is Storj-2 with Storj-1 another target that's barely in use.

 

[Ericloewe]

Okay, so give us the output of zfs get all RaidZ/Storj-2 and zfs get all RaidZ/Storj-1, plus any other datasets doing block storage.

 

[MikeyG]

Definitely just those two set up doing block storage.

Code:

root@freenas:~ # zfs get all RaidZ/Storj-2
NAME		   PROPERTY				 VALUE					SOURCE
RaidZ/Storj-2  type					 volume				   -
RaidZ/Storj-2  creation				 Mon Oct 16 10:55 2017	-
RaidZ/Storj-2  used					 340G					 -
RaidZ/Storj-2  available				8.56T					-
RaidZ/Storj-2  referenced			   340G					 -
RaidZ/Storj-2  compressratio			1.00x					-
RaidZ/Storj-2  reservation			  none					 default
RaidZ/Storj-2  volsize				  2T					   local
RaidZ/Storj-2  volblocksize			 32K					  -
RaidZ/Storj-2  checksum				 on					   default
RaidZ/Storj-2  compression			  lz4					  inherited from							 RaidZ
RaidZ/Storj-2  readonly				 off					  default
RaidZ/Storj-2  copies				   1						default
RaidZ/Storj-2  refreservation		   none					 default
RaidZ/Storj-2  primarycache			 all					  default
RaidZ/Storj-2  secondarycache		   all					  default
RaidZ/Storj-2  usedbysnapshots		  0						-
RaidZ/Storj-2  usedbydataset			340G					 -
RaidZ/Storj-2  usedbychildren		   0						-
RaidZ/Storj-2  usedbyrefreservation	 0						-
RaidZ/Storj-2  logbias				  latency				  default
RaidZ/Storj-2  dedup					off					  default
RaidZ/Storj-2  mlslabel										  -
RaidZ/Storj-2  sync					 standard				 default
RaidZ/Storj-2  refcompressratio		 1.00x					-
RaidZ/Storj-2  written				  340G					 -
RaidZ/Storj-2  logicalused			  342G					 -
RaidZ/Storj-2  logicalreferenced		342G					 -
RaidZ/Storj-2  volmode				  default				  default
RaidZ/Storj-2  snapshot_limit		   none					 default
RaidZ/Storj-2  snapshot_count		   none					 default
RaidZ/Storj-2  redundant_metadata	   all					  default
RaidZ/Storj-2  org.freenas:description  Storj-2				  local
RaidZ/Storj-2  org.freebsd.ioc:active   yes					  inherited from

Code:

root@freenas:~ # zfs get all RaidZ/Storj-1
NAME		   PROPERTY				 VALUE					SOURCE
RaidZ/Storj-1  type					 volume				   -
RaidZ/Storj-1  creation				 Thu Oct 12 16:50 2017	-
RaidZ/Storj-1  used					 18.5G					-
RaidZ/Storj-1  available				8.56T					-
RaidZ/Storj-1  referenced			   18.5G					-
RaidZ/Storj-1  compressratio			1.03x					-
RaidZ/Storj-1  reservation			  none					 default
RaidZ/Storj-1  volsize				  2T					   local
RaidZ/Storj-1  volblocksize			 32K					  -
RaidZ/Storj-1  checksum				 on					   default
RaidZ/Storj-1  compression			  lz4					  inherited from RaidZ
RaidZ/Storj-1  readonly				 off					  default
RaidZ/Storj-1  copies				   1						default
RaidZ/Storj-1  refreservation		   none					 default
RaidZ/Storj-1  primarycache			 all					  default
RaidZ/Storj-1  secondarycache		   all					  default
RaidZ/Storj-1  usedbysnapshots		  0						-
RaidZ/Storj-1  usedbydataset			18.5G					-
RaidZ/Storj-1  usedbychildren		   0						-
RaidZ/Storj-1  usedbyrefreservation	 0						-
RaidZ/Storj-1  logbias				  latency				  default
RaidZ/Storj-1  dedup					off					  default
RaidZ/Storj-1  mlslabel										  -
RaidZ/Storj-1  sync					 standard				 default
RaidZ/Storj-1  refcompressratio		 1.03x					-
RaidZ/Storj-1  written				  18.5G					-
RaidZ/Storj-1  logicalused			  19.1G					-
RaidZ/Storj-1  logicalreferenced		19.1G					-
RaidZ/Storj-1  volmode				  default				  default
RaidZ/Storj-1  snapshot_limit		   none					 default
RaidZ/Storj-1  snapshot_count		   none					 default
RaidZ/Storj-1  redundant_metadata	   all					  default
RaidZ/Storj-1  org.freenas:description  Storj-1				  local
RaidZ/Storj-1  org.freebsd.ioc:active   yes					  inherited from RaidZ

 

[Ericloewe]

With no obvious villains in sight, I guess this comes down to plain old overhead. Unfortunately, estimating usable space with ZFS is far from trivial.

You might get closer to what would be expected using mirrors, but that's a different set of tradeoffs.

 

[MikeyG]

Alright, well thank you for looking at it! As long as I didn't mis-configure something, I guess I'm happy to chalk it up to overhead.