Need help understanding ZFS storage.

[Rickinfl]

Hi,

I need help understanding how FreeNAS does it storage because it seems it eats all your storage up for redundancy.

I have 12 x 8TB drives = 96TB
FreeNAS created a Volume of 87TB
I created a DataStore and says it's total space is 56.1TB

Then I get warnings that my Volume is over 80% my volume is at 81% right now.

So what I'm understanding is zraid and FreeNAS take half my hard drives?
There is redundancy between the 96TB and 87TB now it wants more redundancy? That's almost 50% redundancy from the total TB's installed.

Something just doesn't add up here. Maybe I setup the DataStore or Volumes incorrect?

When do I run out of space? at 87TB or 56.1TB?

So when my Volume hits 81% which is 69TB, but my DataStore is around 47TB and about 82% full I'm out of hard drive space? If that is so I actually run out of hard drive space before my DataStore is full?

 

[garm]

What’s your pool layout? Please post zpool status in code tags.

 

[Rickinfl]

Code:

user@freenas:~ # zpool status 
 pool: DataStore 
state: ONLINE 
 scan: scrub in progress since Sun Apr 15 00:00:02 2018 
		40.9T scanned at 146M/s, 40.9T issued at 146M/s, 72.0T total 
		0 repaired, 56.74% done, 2 days 14:11:52 to go 
config: 

		NAME											STATE	 READ WRITE CKSUM 
		DataStore									   ONLINE	   0	 0	 0 
		  raidz2-0									  ONLINE	   0	 0	 0 
			gptid/01e79ec1-8102-11e7-b2aa-002655e0acaf  ONLINE	   0	 0	 0 
			gptid/02b0a20a-8102-11e7-b2aa-002655e0acaf  ONLINE	   0	 0	 0 
			gptid/037fb075-8102-11e7-b2aa-002655e0acaf  ONLINE	   0	 0	 0 
			gptid/044431df-8102-11e7-b2aa-002655e0acaf  ONLINE	   0	 0	 0 
			gptid/051aba03-8102-11e7-b2aa-002655e0acaf  ONLINE	   0	 0	 0 
			gptid/05e22aa1-8102-11e7-b2aa-002655e0acaf  ONLINE	   0	 0	 0 
		   gptid/06aede56-8102-11e7-b2aa-002655e0acaf  ONLINE	   0	 0	 0 
			gptid/078184f4-8102-11e7-b2aa-002655e0acaf  ONLINE	   0	 0	 0 
			gptid/085a6a35-8102-11e7-b2aa-002655e0acaf  ONLINE	   0	 0	 0 
			gptid/092dc82b-8102-11e7-b2aa-002655e0acaf  ONLINE	   0	 0	 0 
			gptid/09fe40ce-8102-11e7-b2aa-002655e0acaf  ONLINE	   0	 0	 0 
			gptid/0ad4325e-8102-11e7-b2aa-002655e0acaf  ONLINE	   0	 0	 0 

errors: No known data errors 

 pool: freenas-boot 
state: ONLINE 
 scan: scrub repaired 0 in 0 days 00:00:41 with 0 errors on Sat Apr 14 03:45:41 2018 
config: 

		NAME		STATE	 READ WRITE CKSUM 
		freenas-boot  ONLINE	   0	 0	 0 
		  ada0p2	ONLINE	   0	 0	 0

 

[MrToddsFriends]

A RaidZ capacity calculator like this one will answer some of your questions:
http://wintelguy.com/zfs-calc.pl

Enter the following values there (some of them are set per default):

Code:

RAID type:						RAID-Z2 (double parity)
Drive capacity (GB):			  8000
Drive sector size:				4 KiB
Number of drives per RAID group:  12
Number of RAID groups:			1
ZFS record size:				  128 KiB

Take into account:
Check			  Slop space allocation
Check			  20% free space limit

After pressing the calculate button you will find the relevant capacities together with an explanation of these values. I hope this helps.

 

[garm]

https://jsfiddle.net/Biduleohm/paq5u7z5/1/embedded/result/
Will give you the numbers.

 

[danb35]

Hard drives are sold in TB (10^12 bytes), but most OSs measure in TiB (2^40 bytes). 1 TB ~= 0.9 TiB. So your 96 TB are equal to 87 TiB, without accounting for redundancy. Putting all 12 of those disks into a single RAIDZ2 vdev (which isn't a good idea anyway) means you're losing two disks to redundancy--ignoring issues like overhead, reserved space, etc., you're going to have 80 TB, or 72 TiB, of capacity. I can't speak to the "DataStore" you mention, as there's no such thing in ZFS or FreeNAS.

 

[Chris Moore]

Hard drives are sold in TB (10^12 bytes), but most OSs measure in TiB (2^40 bytes). 1 TB ~= 0.9 TiB. So your 96 TB are equal to 87 TiB, without accounting for redundancy. Putting all 12 of those disks into a single RAIDZ2 vdev (which isn't a good idea anyway) means you're losing two disks to redundancy--ignoring issues like overhead, reserved space, etc., you're going to have 80 TB, or 72 TiB, of capacity. I can't speak to the "DataStore" you mention, as there's no such thing in ZFS or FreeNAS.

"DataStore" is the name of the OP's pool ...

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[MrToddsFriends]

The only peculiarity of of ZFS is the "80% rule", which should not come surprisingly for the user as it is noted in the ZFS Primer.

(If one doesn't look closer and such details like slop space allocation get relevant, that is.)

 

[Chris Moore]

When do I run out of space? at 87TB or 56.1TB?

The time to start planning a pool expansion is when you hit the 80% mark, unless you are a proactive person that is always planning ahead. The warning at 80% is there to let you know you are getting close, but when you get to 90% full, the algorithm that ZFS uses to store data changes and the performance will become even slower than it is now.
By the way, your pool is so slow because you made the vdev too wide. I have a server at work with 60 drives in it and it takes about 3 days to do a scrub, where yours has been running since Sunday the 15th and still has 2 days to go. You can add another vdev to your pool to expand capacity or replace all the drives with larger ones. I would suggest that you start planning, but you also need to do some reading because you apparently didn't learn anything about FreeNAS or ZFS before you started shoving a massive amount of data into that system. What is it, are you backing up the whole internet?

Light Reading:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

Don't be afraid to be SAS-sy
https://forums.freenas.org/index.php?resources/don't-be-afraid-to-be-sas-sy.48/

Building, Burn-In, and Testing your FreeNAS system
https://forums.freenas.org/index.php?resources/building-burn-in-and-testing-your-freenas-system.38/

Disk Price/Performance Analysis Buying Information
https://forums.freenas.org/index.ph...e-performance-analysis-buying-information.62/

 

[Rickinfl]

Lol no not backing up the internet. I'm using it for ip cameras. I have 40 cameras writing to it. So I need the max storage.

 

[Chris Moore]

Lol no not backing up the internet. I'm using it for IP cameras. I have 40 cameras writing to it. So I need the max storage.

Any idea how much you will need?
What is your hardware setup?
If you are going to expand the pool, I am happy to give you some suggestions.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Rickinfl]

Setting up a FreeNAS server.
Tyan s7012 mother board dual Xeons 5620
72 Gig of Ram < Will be adding more
2 LSI 9211-8i in IT mode
12 - 8TB hard drives

I need Max space for storage for recording. I don't need a ton of redundancy 2 drives is fine all I'm doing is sending video from 40 Cameras to it for recording for 60 days. They do record in High Def.

Some advice on Hard Drive setup would be great.

 

[Chris Moore]

Some advice on Hard Drive setup would be great.

I went back and looked at some of your previous posts (other threads) and found that back in February you were using a rack chassis. I don't know exactly how long you have been in that but I would suggest moving to a this chassis:
https://www.ebay.com/itm/Server-Chenbro-48-Bay-Top-Loader-4U-Chassis-NEW-in-Box/253503074088
I bought one from this person and he accepted my offer of $350, so I suggest giving it a shot. This chassis can accept 48 drives which will give you room to expand your pool quite a bit before needing to upgrade the chassis again. One of the nice features of this chassis is that it has two, 24 port, SAS expander backplanes which means you can connect one backplane to each port on a single SAS controller using a cable like this:
https://www.ebay.com/itm/Lot-of-2x-...ernal-Mini-SAS-36-Pin-RAID-Cable/152299005370
One of the SAS cards you have is able to control 256 drives and this is how that is done, expander backplanes are fantastic.

If the performance of the pool is adequate, and you just need more space, I would suggest adding another 12 drive vdev to the pool you have. Once you get the existing system moved over to the new chassis, you will have plenty of room for drives. and the WD Purple drives are intended for this type of work load. If you are tolerant of risk, and have a spare or two handy, you might take a chance on desktop drives, which can be less expensive. I use Seagate Desktop drives in my personal, home NAS and have been satisfied with their performance.