What happened to my volume capacity?

[Fraoch]

If I have 4 X 2 TB drives in RAID-Z2, the total capacity should be ~4 TB (2 drives for data, 2 for parity), correct?

After creating a RAID-Z2 pool with these I seem to recall a capacity of 3.5 TB.

Suddenly I have about half that, 1.7 TB? This is not remaining capacity as I add data, this is total capacity. Where did the other 1.7 TB go? I have snapshots turned on but they should show up as used data and I can't see how there would be 1.7 TB of snapshots.

I'm also seeing mysterious, slow writes while the system sits idle. Could be connected?

 

[Ericloewe]

If I have 4 X 2 TB drives in RAID-Z2, the total capacity should be ~4 TB (2 drives for data, 2 for parity), correct?

After creating a RAID-Z2 pool with these I seem to recall a capacity of 3.5 TB.

Suddenly I have about half that, 1.7 TB? This is not remaining capacity as I add data, this is total capacity. Where did the other 1.7 TB go? I have snapshots turned on but they should show up as used data and I can't see how there would be 1.7 TB of snapshots.

My first intuition would be to say 'snapshots', if the data has changed a lot.

I'm also seeing mysterious, slow writes while the system sits idle. Could be connected?

That's probably the System dataset. Nothing to worry about. It handles stuff like Samba 4 non-volatile data.

 

[Fraoch]

My first intuition would be to say 'snapshots', if the data has changed a lot.

Do snapshots show as used data or as a reduction in the total volume capacity?

If it's just a reduction in the total volume capacity that would make sense as some of the iozone tests I'm conducting use large temporary files.

 

[cyberjock]

Actually, I'm wondering if you've set a reservation. That 1.4TB dataset, if it were set as a reservation, would take away from the pool.

The screenshot, as provided, can neither confirm or deny your concern that disk space is "missing". But if you do a zpool status and it shows 7.4TB (or so) for the total pool size then you can safely say that all is fine and that you simply need to figure out what is used and what isn't used. ;)

 

[anodos]

I personally concern myself more with percentages in zpool status than whatever values various tools show.

Snapshots will definitely eat up free space. I hate it when a user decides to rearrange his 600GB folder of random files. My solution is not to hold snapshots long and tell him that kittens die every time he does that.

 

[Fraoch]

Actually, I'm wondering if you've set a reservation. That 1.4TB dataset, if it were set as a reservation, would take away from the pool.

One of the datasets has a reservation. The others, and the volume, don't.

The screenshot, as provided, can neither confirm or deny your concern that disk space is "missing". But if you do a zpool status and it shows 7.4TB (or so) for the total pool size then you can safely say that all is fine and that you simply need to figure out what is used and what isn't used. ;)

zpool status is not showing me total capacity:

Code:

# zpool status
  pool: volume1
 state: ONLINE
  scan: scrub repaired 0 in 19h16m with 0 errors on Sun Oct 26 21:46:58 2014
config:

   NAME  STATE  READ WRITE CKSUM
   volume1  ONLINE  0  0  0
    raidz2-0  ONLINE  0  0  0
    gptid/e8865199-53dd-11e4-a741-0cc47a0984f3  ONLINE  0  0  0
    gptid/bcb83505-53d5-11e4-a741-0cc47a0984f3  ONLINE  0  0  0
    gptid/bd191bf4-53d5-11e4-a741-0cc47a0984f3  ONLINE  0  0  0
    gptid/a3bcd842-5559-11e4-a741-0cc47a0984f3  ONLINE  0  0  0

errors: No known data errors

At any rate, it was snapshots. I turned off snapshots and deleted the ones I had. Now the size is reported as 3.3 TB.

It was an overly-aggressive snapshot setting (every hour) which I set based on the guide, but it now looks more like an example. Every hour would work fine ordinarily but when I'm testing and using iozone to generate several 500 GB files, those snapshots can take up a lot even after the iozone tests are done.

Lesson learned - no snapshots while testing with large files! :D

Thanks for the help everybody.

 

[Mlovelace]

zpool status is not showing me total capacity:

The command that will list capacity is:

Code:

 zfs list -o space 

 

[Fraoch]

The command that will list capacity is:

Code:

 zfs list -o space 

Ah! Thanks.

 

[cyberjock]

Yeah.. I even wrote zpool status and then knew I meant zfs list and apparently didn't fix it. Sorry. I was tired as I had been up 22 hours straight when I wrote that. ;)

Read up on how quotas and reservations work. That will explain it to you. ;)

 

[Fraoch]

Yeah.. I even wrote zpool status and then knew I meant zfs list and apparently didn't fix it. Sorry. I was tired as I had been up 22 hours straight when I wrote that. ;)

LOL, you can be excused.:D Take it easy today.

Read up on how quotas and reservations work. That will explain it to you. ;)

I've been meaning to do that since I don't understand the differences between the two.

 

[cyberjock]

All the more reason not to use them until you understand them. ;)

Using things you don't understand in ZFS can result in loss of pool. I'd *highly* recommend you take a MUCH more conservative outlook when managing ZFS. Data loss is not fun at all. ;)

 

[Fraoch]

I'm only using a quota on one dataset.

I'm viewing any data I have on my pool as disposable until I have sufficiently tested my drives and until I learn more. Situations like I encountered here are quite useful actually, I learned from experience how snapshots impact the total capacity of the pool, not the used capacity. Better to learn this now and you can be sure I'll plan snapshot frequency and duration of retention much more carefully.