Storage space question

Plato · Sep 18, 2017

Hi,

I used for a long time a RAIDZ1 pool with 9 disks ( 6 TB ) and got about 42 TB space.

Now I created a new RAIDZ2 pool with 12 disks ( again 6 TB ) and again I got about the same space as before..

What I don't really understand is how this works? I know there are some calculators out there but even then the disk space should be more I thought.

Even with the metadata and parity how come 10 disk space is reduced to 8 disk space?

Dice · Sep 19, 2017

https://forums.freenas.org/index.php?resources/zfs-raid-size-and-reliability-calculator.49/

Evertb1 · Sep 19, 2017

Plato said:
Hi,

I used for a long time a RAIDZ1 pool with 9 disks ( 6 TB ) and got about 42 TB space.

Now I created a new RAIDZ2 pool with 12 disks ( again 6 TB ) and again I got about the same space as before..

What I don't really understand is how this works? I know there are some calculators out there but even then the disk space should be more I thought.

Even with the metadata and parity how come 10 disk space is reduced to 8 disk space?

Please note that if you compare storage capacities that there is a difference between TB and TiB. That can be the cause of confusion (6 TB is about 5.4xx TiB).

joeschmuck · Sep 19, 2017

Plato said:
Even with the metadata and parity how come 10 disk space is reduced to 8 disk space?

This question seems so basic so I thought I'd toss out a basic answer. In a RAIDZ2, two drives of space are used for parity, this is why you can lose two hard drives and your data is still available. Now that is the simple answer. It is more complicated that that when you start accounting for all the little details like SWAP space, compression, metadata, and more confusing is how ZFS reports it. You almost need to become a ZFS expert to fully understand it. This is not Windows.

Also, I would not worry about minor differences in available space. If FreeNAS is reporting a healthy pool then my advice it to accept it. If you want to become a ZFS guru, there is a ton of data on the internet because ZFS is not proprietary to FreeNAS.

Plato · Sep 19, 2017

Ok, I think I understand. When I was using with 9 disk with RAIDZ1 it said 40 TB used and 2,7 TB remaining. This is about the same capacity as the calculator with recomm. free space + usable data space.

When I enter 12 disk and RAIDZ2 though, it says 42,9 TB (data)+ 10,74 TB (recomm.free space)..

Is it possible that FreeNAS only shows me data portion when I'm not near the full capacity, and when I come close to capacity it shows the total space ( data + recomm.free )? This is the only possible answer as far as I understand. But is this really so?

danb35 · Sep 19, 2017

Plato said:
Is it possible that FreeNAS only shows me data portion when I'm not near the full capacity,

No, FreeNAS doesn't hide capacity like that. Post your pool layout--either the output of zpool status or your Volume Status page from the GUI.

Evertb1 · Sep 19, 2017

Plato said:
Ok, I think I understand. When I was using with 9 disk with RAIDZ1 it said 40 TB used and 2,7 TB remaining. This is about the same capacity as the calculator with recomm. free space + usable data space.

When I enter 12 disk and RAIDZ2 though, it says 42,9 TB (data)+ 10,74 TB (recomm.free space)..

Is it possible that FreeNAS only shows me data portion when I'm not near the full capacity, and when I come close to capacity it shows the total space ( data + recomm.free )? This is the only possible answer as far as I understand. But is this really so?

According to you you had a RaidZ1 with 9 drives of 6 TB and now you have a RAIDZ2 with 12 x 6 TB
According to the calculator you should get this figures:
RAIDZ1 9 x 6 TB = 43.66 TiB (data space)
RAIDZ2 12 x 6 TB = 54.57 TiB (data space)

The used space and free space of your RAIDZ1 storage is close enough to the calculated data space, but you are way below the recommended free space. If you would have stayed within the recommendation you would have counted with 34.37 TiB usable data space. Meaning that you are late with expanding your storage.

With your new RAIDZ2 storage you will end with 42.96 usable data space if you take the recommended free space in account. That is more then 8 TiB bigger then the usable space on your old storage. But you already have growth problems if you want to store the data from the old storage on it, because you already have 40 TB of data.

Plato · Sep 19, 2017

danb35 said:
No, FreeNAS doesn't hide capacity like that. Post your pool layout--either the output of zpool status or your Volume Status page from the GUI.

Here is the zpool list -v output:

Code:

root@freenas:~ # zpool list -v media2
NAME									 SIZE ALLOC FREE EXPANDSZ FRAG	CAP DEDUP HEALTH ALTROOT
media2									65T 18.8T 46.2T		 -	12%	28% 1.00x ONLINE /mnt
 raidz2								 65T 18.8T 46.2T		 -	12%	28%
 gptid/fe490f59-955b-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
 gptid/ff3adb39-955b-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
 gptid/002475d0-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
 gptid/01155a65-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
 gptid/020cb428-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
 gptid/02fed389-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
 gptid/03f2e5ed-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
 gptid/04f2c8d1-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
 gptid/05f52a22-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
 gptid/06f50bef-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
 gptid/07f5b245-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
 gptid/08f8f51e-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -

Here is the zpool status output:

Code:

 
 pool: media2 
 state: ONLINE
 scan: none requested
 config: 
 NAME											STATE	 READ WRITE CKSUM 
 media2										 ONLINE	 0	 0	 0 
 raidz2-0									 ONLINE	 0	 0	 0 
 gptid/fe490f59-955b-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0 
 gptid/ff3adb39-955b-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0 
 gptid/002475d0-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0 
 gptid/01155a65-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0 
 gptid/020cb428-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0 
 gptid/02fed389-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0 
 gptid/03f2e5ed-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0 
 gptid/04f2c8d1-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0 
 gptid/05f52a22-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0 
 gptid/06f50bef-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0 
 gptid/07f5b245-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0 
 gptid/08f8f51e-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0

Code:

root@freenas:~ # df -h media2
Filesystem	Size	Used Avail Capacity Mounted on
media2		 42T	8.3T	 34T	20%	/mnt/media2

danb35 · Sep 19, 2017

Plato said:
media2 65T 18.8T 46.2T - 12% 28% 1.00x ONLINE /mnt

So this is as expected--72 TB (total capacity of your 12 disks--though you really shouldn't be using vdevs that wide) is 65 TiB, of which 18.8 TiB are used (including parity) and 46.2 TiB are available (again, including space for parity).

In the web GUI, what does the storage screen say? Specifically, what's the second line for media2?

Evertb1 · Sep 19, 2017

Plato said:

Here is the zpool list -v output:

Code:

root@freenas:~ # zpool list -v media2
NAME									 SIZE ALLOC FREE EXPANDSZ FRAG	CAP DEDUP HEALTH ALTROOT
media2									65T 18.8T 46.2T		 -	12%	28% 1.00x ONLINE /mnt
raidz2								 65T 18.8T 46.2T		 -	12%	28%
gptid/fe490f59-955b-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
gptid/ff3adb39-955b-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
gptid/002475d0-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
gptid/01155a65-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
gptid/020cb428-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
gptid/02fed389-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
gptid/03f2e5ed-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
gptid/04f2c8d1-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
gptid/05f52a22-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
gptid/06f50bef-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
gptid/07f5b245-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -
gptid/08f8f51e-955c-11e7-8bbb-d05099c089a6	 -	 -	 -		 -	 -	 -

Here is the zpool status output:

Code:

pool: media2
state: ONLINE
scan: none requested
config:
NAME											STATE	 READ WRITE CKSUM
media2										 ONLINE	 0	 0	 0
raidz2-0									 ONLINE	 0	 0	 0
gptid/fe490f59-955b-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0
gptid/ff3adb39-955b-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0
gptid/002475d0-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0
gptid/01155a65-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0
gptid/020cb428-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0
gptid/02fed389-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0
gptid/03f2e5ed-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0
gptid/04f2c8d1-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0
gptid/05f52a22-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0
gptid/06f50bef-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0
gptid/07f5b245-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0
gptid/08f8f51e-955c-11e7-8bbb-d05099c089a6 ONLINE	 0	 0	 0

Code:

root@freenas:~ # df -h media2
Filesystem	Size	Used Avail Capacity Mounted on
media2		 42T	8.3T	 34T	20%	/mnt/media2

I see an allocation of 18 TB. Looks a bit high to me. I expected to see an allocation of 10 - 11 TB. The size (65 TB) equals about the calculated RAID space. Taking the available space and allocated space in account your Volume status is OK I think but maybe one of the brighter lights in this forum have an explanation for that high allocation. I have a far bit smaller storage (5 x 2 TB RAIDZ2) but the size of the allocation is about 16 % of the available storage. Your allocation is something like 27%.

danb35 · Sep 19, 2017

Evertb1 said:
I see an allocation of 18 TB. Looks a bit high to me.

Why do you say this? The amount allocated is simply the amount of space used, and will be larger or smaller depending on how much data is in the pool. Here's mine:

Code:

➜  ~ zpool list tank
NAME  SIZE  ALLOC  FREE  EXPANDSZ  FRAG  CAP  DEDUP  HEALTH  ALTROOT
tank  65.1T  38.7T  26.4T  -  31%  59%  1.04x  ONLINE  /mnt

Evertb1 · Sep 19, 2017

danb35 said:
Why do you say this? The amount allocated is simply the amount of space used, and will be larger or smaller depending on how much data is in the pool. Here's mine:

Code:
➜ ~ zpool list tank NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT tank 65.1T 38.7T 26.4T - 31% 59% 1.04x ONLINE /mnt

I did not see that OP already had worth of 8 TB data stored. If that was not the case the allocated space would all be parity and overhead. And that in my book was a bit much.

Plato · Sep 19, 2017

Evertb1 said:
I did not see that OP already had worth of 8 TB data stored. If that was not the case the allocated space would all be parity and overhead. And that in my book was a bit much.

Yes, that's the case. Only 8 TB was used disk space but it shows 18 TB as allocated.

When I mount the share to the windows it looks better though not much. It shows 47,9 TB as total disk space. Why it shows different than FreeBSD I'm not sure though...

Web gui report shows same values as df 33,6 TB free and 8,3 TB used.

Johnnie Black · Sep 19, 2017

Just FYI, I find this calculator gives a much more accurate usable space because it accounts for padding and slop space:

http://wintelguy.com/zfs-calc.pl

In your case, 12 x 6TB RAIDZ2 = 47.97TiB total usable, same as windows reports, though Windows displays TB when it should display TiB

Plato · Sep 19, 2017

Yes it seems this calculator is closer to what I see from Windows. For parity a loss of ~%17 is usual but as I see there is also a loss due to padding ( metadata ? ) and it's about ~%7 which is huge when the drives used are big.

What windows shows is true though, because TiB is actually a term only used to differentiate from the TB which market uses on storage technology. Operating systems, uses TB as 2^40.

MrToddsFriends · Sep 19, 2017

Plato said:
What windows shows is true though, because TiB is actually a term only used to differentiate from the TB which market uses on storage technology. Operating systems, uses TB as 2^40.

The use of binary and decimal prefixes is internationally standardized.
https://physics.nist.gov/cuu/Units/prefixes.html
https://physics.nist.gov/cuu/Units/binary.html (approved by the IEC for use in the fields of data processing and data transmission in 1998)

In some places decimal prefixes misleadingly are still used, where binary ones would be correct.

Important Announcement for the TrueNAS Community.

Storage space question

Plato

Contributor

Dice

Wizard

Evertb1

Guru

joeschmuck

Old Man

Plato

Contributor

danb35

Hall of Famer

Evertb1

Guru

Plato

Contributor

danb35

Hall of Famer

Evertb1

Guru

danb35

Hall of Famer

Evertb1

Guru

Plato

Contributor

Johnnie Black

Guru

Plato

Contributor

MrToddsFriends

Documentation Browser

Similar threads