Is it possible to disable space reservation for zvol's snapshot?

[jakubjb]

I know, that normal ZFS behavior is to reserve space for zvol's snapshot, up to actual usedbydataset. Example from my server:

Code:

NAME                        PROPERTY              VALUE                  SOURCE
pool/zvol-iscsi-1-a  used                  17.2T                  -
pool/zvol-iscsi-1-a  volsize               10T                    local
pool/zvol-iscsi-1-a  usedbydataset         6.88T                  -

One snapshot exists:

Code:

NAME                                               USED  AVAIL  REFER  MOUNTPOINT
pool/zvol-iscsi-1-a@auto-20170215.1308-1h  57.9M      -  6.88T  -

Is it possible to disable this behavior? If not, what is the chance for succesful feature request?

I'm asking for opinion of more experienced requesters, it seems to me that FreeBSD/ZFS developers are quite strict and intimidating (not an offence, just my perception after reading lots of docs).

 

[bigphil]

I'm a little confused here, but what I think you might be looking for is a "sparse volume." By default, refreservation is set to equal the size of your zvol. You create a 1TB zvol, refreservation is set to that. If refreservation is set, a snapshot is only allowed if there is enough free pool space outside of this reservation to accommodate the current number of referenced bytes in the dataset. Some additional reading for you here.

 

[jakubjb]

Thanks for reply. I want to be sure we're on the same page.
According to my example, if I had this 10 TB zvol created as sparse volume, then after snapshoting it would take 7 TB of actual used space plus 7 TB reserved?
So, the "used" property would be 14 TB instead of 17 TB that it's now?
The behavior in this case is the same, the only difference is lack of initial reservation for non-sparse volume. That's clear to me.

But how to avoid the reservation of 7 TB at the moment of taking a snapshot of volume? As I understand, if I set quota lower then usedbydataset plus required reservation, then I won't be able to snapshot, right?

 

[bigphil]

Post the output of "zfs get all pool/zvol-iscsi-1-a" so we can better understand what you're working with.

 

[jakubjb]

Here it is:

Code:

NAME                        PROPERTY              VALUE                  SOURCE
pool/zvol-iscsi-1-a  type                  volume                 -
pool/zvol-iscsi-1-a  creation              Mon Sep 26  9:54 2016  -
pool/zvol-iscsi-1-a  used                  17.2T                  -
pool/zvol-iscsi-1-a  available             23.1T                  -
pool/zvol-iscsi-1-a  referenced            6.88T                  -
pool/zvol-iscsi-1-a  compressratio         1.09x                  -
pool/zvol-iscsi-1-a  reservation           none                   default
pool/zvol-iscsi-1-a  volsize               10T                    local
pool/zvol-iscsi-1-a  volblocksize          128K                   -
pool/zvol-iscsi-1-a  checksum              fletcher2              inherited from pool
pool/zvol-iscsi-1-a  compression           lz4                    local
pool/zvol-iscsi-1-a  readonly              off                    default
pool/zvol-iscsi-1-a  copies                1                      default
pool/zvol-iscsi-1-a  refreservation        10.3T                  local
pool/zvol-iscsi-1-a  primarycache          all                    inherited from pool
pool/zvol-iscsi-1-a  secondarycache        all                    default
pool/zvol-iscsi-1-a  usedbysnapshots       33.9M                  -
pool/zvol-iscsi-1-a  usedbydataset         6.88T                  -
pool/zvol-iscsi-1-a  usedbychildren        0                      -
pool/zvol-iscsi-1-a  usedbyrefreservation  10.3T                  -
pool/zvol-iscsi-1-a  logbias               latency                default
pool/zvol-iscsi-1-a  dedup                 off                    default
pool/zvol-iscsi-1-a  mlslabel                                     -
pool/zvol-iscsi-1-a  sync                  standard               default
pool/zvol-iscsi-1-a  refcompressratio      1.09x                  -
pool/zvol-iscsi-1-a  written               23.4M                  -
pool/zvol-iscsi-1-a  logicalused           7.43T                  -
pool/zvol-iscsi-1-a  logicalreferenced     7.43T                  -
pool/zvol-iscsi-1-a  volmode               default                default
pool/zvol-iscsi-1-a  snapshot_limit        none                   default
pool/zvol-iscsi-1-a  snapshot_count        none                   default
pool/zvol-iscsi-1-a  redundant_metadata    all                    default

Snapshots:

Code:

NAME                                               USED  AVAIL  REFER  MOUNTPOINT
pool/zvol-iscsi-1-a@auto-20170215.1908-1h  33.9M      -  6.88T  -

What I don't like is "wasting" of additional ("used" - "volsize" = "usedbydataset") TB of space when there are snapshots taken.
Before taking the snapshot, pool had 19 TB of space "available", now it's 12 TB.
As far as I know it's by design, but if there's a way to disable such behavior, I'll be more then happy.

 

[bigphil]

Before taking the snapshot, pool had 19 TB of space "available", now it's 12 TB

But the output shows 23.1T available in the pool? A snapshot isn't going to just immediately consume 7TB of space. There is something else going on here. Are there any child datasets that have any reservations or quotas set or clones? Output shows 17.2T used, but this dataset only shows 6.88T used.

 

[jakubjb]

I'll do the exercise once again and show the changes to pool and zvol properties, before and after. Bear with me, please.

Edit: Unfortunatelly "Held by replication system" message shows up after I try to remove snapshot. Periodic snapshot task is removed, replication task is removed. I even manually removed hold on snapshot. I'll give some time to GUI to catch up or destroy snapshot manually.
As they say, the deeper into the forest, the more trees ;-)

 

[jakubjb]

I'm back, I needed to perform some manual operations.
Pool and zvols with no zvol's snapshots:

Code:

NAME                                            USED  AVAIL  REFER  MOUNTPOINT
...
pool                                            86.9T  19.6T   571M  /mnt/pool
...
pool/zvol-iscsi-1-a                             10.3T  23.1T  6.84T  -
pool/zvol-iscsi-2-a                             12.4T  22.9T  9.09T  -
pool/zvol-iscsi-3                               5.16T  23.2T  1.57T  -
pool/zvol-iscsi-4                               5.16T  21.2T  3.57T  -
...
pool/ozvol-iscsi                                  1.03T  20.1T   561G  -
...
pool/zvol-iscsi                                   217G  19.7T   155G  -
pool/zvol-iscsi-model                             224G  19.8T  17.4G  -
pool/zvol-iscsi-produkcja                        2.30T  21.7T   243G  -
...

As You can see all zvols have AVAIL > pool's AVAIL. The difference is - at least it's mathematically OK, approximately - (zvol's: VOLSIZE - REFERENCED). I find it somehow logical.
Let's take a look at one of zvols:

Code:

NAME                        PROPERTY              VALUE                  SOURCE
pool/zvol-iscsi-1-a  type                  volume                 -
pool/zvol-iscsi-1-a  creation              Mon Sep 26  9:54 2016  -
pool/zvol-iscsi-1-a  used                  10.3T                  -
pool/zvol-iscsi-1-a  available             23.1T                  -
pool/zvol-iscsi-1-a  referenced            6.83T                  -
pool/zvol-iscsi-1-a  compressratio         1.09x                  -
pool/zvol-iscsi-1-a  reservation           none                   default
pool/zvol-iscsi-1-a  volsize               10T                    local
pool/zvol-iscsi-1-a  volblocksize          128K                   -
pool/zvol-iscsi-1-a  checksum              fletcher2              inherited from pool
pool/zvol-iscsi-1-a  compression           lz4                    local
pool/zvol-iscsi-1-a  readonly              off                    default
pool/zvol-iscsi-1-a  copies                1                      default
pool/zvol-iscsi-1-a  refreservation        10.3T                  local
pool/zvol-iscsi-1-a  primarycache          all                    inherited from pool
pool/zvol-iscsi-1-a  secondarycache        all                    default
pool/zvol-iscsi-1-a  usedbysnapshots       0                      -
pool/zvol-iscsi-1-a  usedbydataset         6.83T                  -
pool/zvol-iscsi-1-a  usedbychildren        0                      -
pool/zvol-iscsi-1-a  usedbyrefreservation  3.48T                  -
pool/zvol-iscsi-1-a  logbias               latency                default
pool/zvol-iscsi-1-a  dedup                 off                    default
pool/zvol-iscsi-1-a  mlslabel                                     -
pool/zvol-iscsi-1-a  sync                  standard               default
pool/zvol-iscsi-1-a  refcompressratio      1.09x                  -
pool/zvol-iscsi-1-a  written               6.83T                  -
pool/zvol-iscsi-1-a  logicalused           7.39T                  -
pool/zvol-iscsi-1-a  logicalreferenced     7.39T                  -
pool/zvol-iscsi-1-a  volmode               default                default
pool/zvol-iscsi-1-a  snapshot_limit        none                   default
pool/zvol-iscsi-1-a  snapshot_count        none                   default
pool/zvol-iscsi-1-a  redundant_metadata    all                    default

USED = VOLSIZE = 10 TiB.

Now let's take a snapshot:

Code:

NAME                                        USED  AVAIL  REFER  MOUNTPOINT
pool/zvol-iscsi-1-a@auto-20170216.1205-1h      0      -  6.83T  -

What changed on pool on zvols:

Code:

NAME                                            USED  AVAIL  REFER  MOUNTPOINT
...
pool                                            93.7T  12.8T   571M  /mnt/pool
...
pool/zvol-iscsi-1-a                             17.1T  23.1T  6.83T  -
pool/zvol-iscsi-2-a                             12.4T  16.1T  9.10T  -
pool/zvol-iscsi-3                               5.16T  16.4T  1.57T  -
pool/zvol-iscsi-4                               5.16T  14.4T  3.58T  -
...
pool/ozvol-iscsi                                  1.03T  13.3T   561G  -
...
pool/zvol-iscsi                                   217G  12.9T   155G  -
pool/zvol-iscsi-model                             224G  13.0T  17.4G  -
pool/zvol-iscsi-produkcja                        2.30T  14.9T   243G  -
...

Nothing changed for zvol with snapshot, but pool and all other zvols AVAIL decreased, approximately by amount equal to REFERENCED space of zvol with snapshot.
Finally, zvol with snapshot:

Code:

NAME                        PROPERTY              VALUE                  SOURCE
pool/zvol-iscsi-1-a  type                  volume                 -
pool/zvol-iscsi-1-a  creation              Mon Sep 26  9:54 2016  -
pool/zvol-iscsi-1-a  used                  17.1T                  -
pool/zvol-iscsi-1-a  available             23.1T                  -
pool/zvol-iscsi-1-a  referenced            6.83T                  -
pool/zvol-iscsi-1-a  compressratio         1.09x                  -
pool/zvol-iscsi-1-a  reservation           none                   default
pool/zvol-iscsi-1-a  volsize               10T                    local
pool/zvol-iscsi-1-a  volblocksize          128K                   -
pool/zvol-iscsi-1-a  checksum              fletcher2              inherited from pool
pool/zvol-iscsi-1-a  compression           lz4                    local
pool/zvol-iscsi-1-a  readonly              off                    default
pool/zvol-iscsi-1-a  copies                1                      default
pool/zvol-iscsi-1-a  refreservation        10.3T                  local
pool/zvol-iscsi-1-a  primarycache          all                    inherited from pool
pool/zvol-iscsi-1-a  secondarycache        all                    default
pool/zvol-iscsi-1-a  usedbysnapshots       0                      -
pool/zvol-iscsi-1-a  usedbydataset         6.83T                  -
pool/zvol-iscsi-1-a  usedbychildren        0                      -
pool/zvol-iscsi-1-a  usedbyrefreservation  10.3T                  -
pool/zvol-iscsi-1-a  logbias               latency                default
pool/zvol-iscsi-1-a  dedup                 off                    default
pool/zvol-iscsi-1-a  mlslabel                                     -
pool/zvol-iscsi-1-a  sync                  standard               default
pool/zvol-iscsi-1-a  refcompressratio      1.09x                  -
pool/zvol-iscsi-1-a  written               0                      -
pool/zvol-iscsi-1-a  logicalused           7.39T                  -
pool/zvol-iscsi-1-a  logicalreferenced     7.39T                  -
pool/zvol-iscsi-1-a  volmode               default                default
pool/zvol-iscsi-1-a  snapshot_limit        none                   default
pool/zvol-iscsi-1-a  snapshot_count        none                   default
pool/zvol-iscsi-1-a  redundant_metadata    all                    default

USED is no more equal to VOLSIZE, it's bigger by, approximately, REFERENCED (or USEDBYDATASET) of this zvol.

That was to be proved. When You take a snapshot, ZFS reserves space which let's You completely rewrite data currently existing on zvol. I hope it's clearly visible above.

My question still remains: can I disable this behavior?

 

[bigphil]

Like I said in my first post...use a sparse volume and you shouldn't see this behavior.

 

[jakubjb]

Workaround, not a solution... but I'll give it a try. I hope it's gonna work differently. Thanks, bigphil.

 

[jakubjb]

Quick test and I confirmed that taking a snapshot of sparse volume does not reserve any additional space.
I guess that while I see "NOT RECOMENDED" comment, next to "sparse volume" option in FreeNAS GUI, it's because using sparse volumes may lead to over provisioning without even noticing it. I probably need to deep dive into eventual performance issues with this kind of volume.
Thanks again, case closed.

 

[bigphil]

I guess that while I see "NOT RECOMENDED" comment, next to "sparse volume" option in FreeNAS GUI, it's because using sparse volumes may lead to over provisioning without even noticing it.

Exactly. Depending on the host type consuming the volume this can be an issue or not. For instance, FreeNAS supports the VMware ESXi VAAI primitive for tp stun/out of space condition and can notify you of when there may be a space issue by configurable settings on the iSCSI global config or extent config. So...depending on what type of host is consuming your volume, these issues can be less worrisome.