Having trouble understanding space usage

[R H]

24 x 6TB SAS drives

FreeNAS reports that the volume is over 100TB:

I have an extent that I created that is 80TB:

However, it shows up on the system as 70TB:

It is connected to a Server 2012 system via iSCSI, which reports used space at 56.8TB and free space as 23.1 TB:

And FreeNAS reports that Vol_1 is 92% full.

I'm afraid that when they get to 70TB, something's going to break... and that time is fast approaching.

Can someone help me figure out what's going on here?

The only file in /mnt/Vol_1 is the extent file of 67TB.

 

[Ericloewe]

A few things:

The top level is raw disk space. Some of that will be used for parity.

The second level is the usable space, the top-level dataset. It's recommended that you stick stuff into lower-level datasets, by the way.

If you're using iSCSI, FreeNAS only sees a big-ass file (or similar). Not how much is free, all of it.

This big-ass file is wayyy too big-ass for your pool. If using iSCSI, you should not fill it up more than 50-60%. When not using iSCSI, it should not be filled up more than 80%.

Regarding the 70TB vs 80TB, my guess is that FreeNAS limited it to something the pool could (barely and badly) store.

 

[R H]

Wow.... 50%?

That sounds like a real waste.

This ALL sounds like a waste.

So... how do I fix this? What is there to do at this point?

Edit: According to this math-- I can put 144TB of RAW space in, get ~65TB of "usable" space that I can only use about 32TB of? Why?

 

[Ericloewe]

Wow.... 50%?

That sounds like a real waste.

This ALL sounds like a waste.

So... how do I fix this? What is there to do at this point?

Add more storage or remove some data.

Honestly, at this point, I recommend you do some reading around the forum, as using iSCSI has a few additional pitfalls that need to be navigated around. iSCSI really is that option you only want to use if you can use a normal share.

 

[Darren Myers]

I was in the same boat as you, looking at FreeNAS to use as a iSCSI target...until i read the 50% and i stopped my planning and stuck with CIFS and called it a day so i can at *least* use 70-80% of my pools.

Edit: According to this math-- I can put 144TB of RAW space in, get ~65TB of "usable" space that I can only use about 32TB of? Why?

I was told this has to do with how when using iSCSI and ZFS together it causes fragmentation issues , and the 50-60% that is the maximum recommended to fill is so ZFS can manage that fragmentation; its possible @cyberjock can weigh in more if he has time , but i *thinkkk* i got it.

 

[cyberjock]

Nothing to add. What you said Darren is basically how it is. :P

 

[Robert Trevellyan]

According to this math-- I can put 144TB of RAW space in, get ~65TB of "usable" space that I can only use about 32TB of? Why?

This thread may help: Why iSCSI often requires more resources for the same result

 

[R H]

I've been reading these articles, but they don't make any sense to me, insofar as iSCSI has had great performance for me - 6 - 7x the performance of a CIFS share from the same FreeNAS box. I can get a gigabit, maybe 2 when using FreeNAS CIFS, but when I mount an iSCSI share from FreeNAS, I can get between 7gbits to pretty close to a full 10 gigabits writing over the network.

We have extraordinarily simple needs... but I can't seem to get the formula right. I thought I had it with iSCSI because the performance was great... but now I doubt my own veracity.

I have a bunch of disks and I need to share them out on the network... it's a couple of supermicro chassis with 12gbps backplanes-- one chassis has a motherboard running FreeNAS and a 10GB network card that gets along great with our network based on my iSCSI speeds. It sees all of the disks in both chassis properly. I just need to share them on a windows network, but also have the disk parity / redundancy that z3 provides.

Is there a reason that CIFS is failing to provide the same speed?

EDIT: I have a unique opportunity because I *just* put the second chassis of disks online so there's a good chance I can change my strategy right now. I'd love to have any input from previous experience anyone can provide.

 

[mjws00]

CIFS is single threaded per connection. Even a fast box gets processor bound ~300MB/s on a single link, lots of overhead to contend with. You are in the ball park. If you can get more threads working you should get more bandwidth.

7 Gbps on an Intel card via iscsi is pretty much right on target as well, in fact better than average. No idea what you are serving up, but Z3 can be pretty slow.