cannot create snapshot - out of space

[linux203]

I found a similar thread, but the person only had ~15MB free in the pool. I have nearly 300GB free to snapshot a 30GB dataset with only 2.2GB used. What am I missing?

FreeNAS-8.0.4-RELEASE-p1-x64 (11059)

[root@miwfdfsu04] ~# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
disk1 1.80T 1.35T 462G 74% ONLINE /mnt
disk2 696G 294G 402G 42% ONLINE /mnt

[root@miwfdfsu04] ~# zfs list -o space
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
disk1 297G 1.03T 0 361G 0 698G
disk1/grpwise 27.8G 30G 0 2.20G 27.8G 0
disk1/miwfdasl03 82.4G 668G 0 668G 0 0
disk2 387G 298G 0 25.9G 0 272G
disk2/backup 46.8G 17.2G 0 17.2G 0 0
disk2/miwfdwwl03 53.6G 50G 0 46.4G 3.62G 0
disk2/vm_images 48.4G 202G 0 202G 0 0
disk2/vm_iso 147G 3.22G 0 3.22G 0 0

[root@miwfdfsu04] ~# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/FreeNASs1a 927M 379M 474M 44% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/md0 4.6M 1.8M 2.3M 44% /etc
/dev/md1 824K 2.5K 756K 0% /mnt
/dev/md2 149M 10M 127M 7% /var
/dev/ufs/FreeNASs4 20M 2.9M 15M 16% /data
disk2 413G 26G 387G 6% /mnt/disk2
disk2/backup 64G 17G 47G 27% /mnt/disk2/backup
disk2/miwfdwwl03 100G 46G 54G 46% /mnt/disk2/miwfdwwl03
disk2/vm_images 250G 202G 48G 81% /mnt/disk2/vm_images
disk2/vm_iso 150G 3.2G 147G 2% /mnt/disk2/vm_iso
disk1 657G 361G 297G 55% /mnt/disk1
disk1/miwfdasl03 750G 668G 82G 89% /mnt/disk1/miwfdasl03
/dev/md3 1.9G 3.4M 1.7G 0% /var/tmp/.cache
disk1/grpwise 30G 2.2G 28G 7% /mnt/disk1/grpwise


[root@miwfdfsu04] ~# zfs snapshot disk1/grpwise@20120426-grpwise
cannot create snapshot 'disk1/grpwise@20120426-grpwise': out of space

 

[krs]

I have the same Issue

FreeNAS-8.0.4-RELEASE-p2-x86 (11367)

zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
backup 1.82T 246G 1.58T 13% ONLINE /mnt

zfs list -o space
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
backup 1.38T 420G 0 156K 0 420G
backup/share 1.39T 100G 0 92.6G 7.42G 0
backup/tm-david 27.0G 160G 0 153G 6.96G 0
backup/tm-filippo 180G 100G 0 144K 100G 0
backup/tm-simon 120G 60G 0 144K 60.0G 0


df -h
Filesystem Size Used Avail Capacity Mounted on
backup 1.4T 156K 1.4T 0% /mnt/backup
backup/share 1.5T 93G 1.4T 6% /mnt/backup/share
backup/tm-david 180G 153G 27G 85% /mnt/backup/tm-david
backup/tm-filippo 180G 144K 180G 0% /mnt/backup/tm-filippo
backup/tm-simon 120G 144K 120G 0% /mnt/backup/tm-simon

 

[William Grzybowski]

zfs list -t snapshot

 

[krs]

~# zfs list -t snapshot
no datasets available

~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
backup 420G 1.38T 156K /mnt/backup
backup/share 100G 1.39T 92.6G /mnt/backup/share
backup/tm-david 160G 26.6G 153G /mnt/backup/tm-david
backup/tm-filippo 100G 180G 144K /mnt/backup/tm-filippo
backup/tm-simon 60G 120G 144K /mnt/backup/tm-simon

I also cannot add a new dataset (even a 150G one)

 

[William Grzybowski]

Do you have quota, refquote, reservation or refreservation set for any of those?

# zfs list -o name,quota,refquota,reservation,refreservation

 

[krs]

Yes :
zfs list -o name,quota,refquota,reservation,refreservation
NAME QUOTA REFQUOTA RESERV REFRESERV
backup none none none none
backup/share none none none 100G
backup/tm-david none 180G none 160G
backup/tm-filippo none 180G none 100G
backup/tm-simon none 120G none 60G

But I have enough free and unreserved space to add another dataset, despite that I get a "not enough space" error when I try.

 

[William Grzybowski]

Thats fault of the refreservation, obviously.
You've reserved all of your space (420G) to those 4 datasets, it is like those datasets were full, thats the whole of refreservation.

 

[krs]

But I have 1.38T available on my main volume.

 

[William Grzybowski]

Right, I have misread the output, sorry.

I can't tell why then... I would try to remove the refreservation just to make sure it is unrelated, even though you have enough free space.

 

[krs]

Good advice, I removed the refreservations and now I'm able to create new datasets and snapshots. Must be a bug somewhere... anyway, thank you for your time!

 

[human]

Hi,
I just thought I'd bump this in case anyone wants to look at it for me... If this is just me not knowing what I'm doing, please advise. Otherwise, I'll go ahead and submit a bug report. But I thought I'd ask in the forum first...

The issue I'm reporting here seems to be that when the system tries to create a snapshot, it reports "out of space" when it does not seem to be out of space.

This seems to be caused because quotas and/or reservations are set on the datasets.

my system is:
Build FreeNAS-9.2.1.5-RELEASE-x64 (80c1d35)
Platform Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
Memory 16249MB

In setting up a new FreeNAS server for the first time (I don't completely know what the hell I'm doing) I noticed that my terminal output in my web GUI footer soon started filling up with this, repeated once every minute:

May 25 18:41:00 freenas autosnap.py: [tools.autosnap:58] Popen()ing: /sbin/zfs snapshot first-volume/main-dataset@auto-20140525.1841-6w
May 25 18:41:00 freenas autosnap.py: [tools.autosnap:241] Failed to create snapshot 'first-volume/main-dataset@auto-20140525.1841-6w': cannot create snapshot 'first-volume/main-dataset@auto-20140525.1841-6w': out of space

May 25 18:42:01 freenas autosnap.py: [tools.autosnap:58] Popen()ing: /sbin/zfs snapshot first-volume/main-dataset@auto-20140525.1842-6w
May 25 18:42:01 freenas autosnap.py: [tools.autosnap:241] Failed to create snapshot 'first-volume/main-dataset@auto-20140525.1842-6w': cannot create snapshot 'first-volume/main-dataset@auto-20140525.1842-6w': out of space

May 25 18:43:01 freenas autosnap.py: [tools.autosnap:58] Popen()ing: /sbin/zfs snapshot first-volume/main-dataset@auto-20140525.1843-6w
May 25 18:43:01 freenas autosnap.py: [tools.autosnap:241] Failed to create snapshot 'first-volume/main-dataset@auto-20140525.1843-6w': cannot create snapshot 'first-volume/main-dataset@auto-20140525.1843-6w': out of space

And indeed, no snapshots have been created. The system is unable to create the scheduled snapshot.

Code:

Here is a picture of my system BEFORE I removed quotas and reservations:
 
[root@freenas ~]# zfs list -t snapshot                                     
no datasets available                                                       
[root@freenas ~]#
 
[root@freenas ~]# zfs list                                                 
NAME                          USED  AVAIL  REFER  MOUNTPOINT               
first-volume                1.49T  2.08T  168K  /mnt/first-volume         
first-volume/.system        1.08M  2.08T  168K  /mnt/first-volume/.system 
first-volume/.system/cores    144K  2.08T  144K  /mnt/first-volume/.system/cores
first-volume/.system/samba4  412K  2.08T  412K  /mnt/first-volume/.system/samba4
first-volume/.system/syslog  380K  2.08T  380K  /mnt/first-volume/.system/syslog
first-volume/first-dataset      1T  1024G  144K  /mnt/first-volume/first-dataset
first-volume/main-dataset    500G  497G  2.99G  /mnt/first-volume/main-dataset
[root@freenas ~]#
 
 
[root@freenas ~]# zfs list -o name,quota,refquota,reservation,refreservation
NAME                        QUOTA  REFQUOTA  RESERV  REFRESERV             
first-volume                  none      none    none      none             
first-volume/.system          none      none    none      none             
first-volume/.system/cores    none      none    none      none             
first-volume/.system/samba4  none      none    none      none             
first-volume/.system/syslog  none      none    none      none             
first-volume/first-dataset      1T        1T    none        1T             
first-volume/main-dataset    500G      500G    none      500G             
[root@freenas ~]#
 
 
 
And this is what it looks like AFTER removing quotas and reservations:
 
[root@freenas ~]# zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT first-volume/main-dataset@auto-20140525.2041-6w 0 - 2.99G - [root@freenas ~]#  

[root@freenas ~]#   
[root@freenas ~]# zfs list
NAME                          USED  AVAIL  REFER  MOUNTPOINT 
first-volume                3.00G  3.57T  168K  /mnt/first-volume
first-volume/.system        1.11M  3.57T  168K  /mnt/first-volume/.system       
first-volume/.system/cores    144K  3.57T  144K  /mnt/first-volume/.system/cores
first-volume/.system/samba4  412K  3.57T  412K  /mnt/first-volume/.system/samba4       
first-volume/.system/syslog  408K  3.57T  408K  /mnt/first-volume/.system/syslog       
first-volume/first-dataset    144K  3.57T  144K  /mnt/first-volume/first-dataset         
first-volume/main-dataset    2.99G  3.57T  2.99G  /mnt/first-volume/main-dataset         
[root@freenas ~]#                                                 
[root@freenas ~]# zfs list -o name,quota,refquota,reservation,refreservation           
NAME                        QUOTA  REFQUOTA  RESERV  REFRESERV                 
first-volume                  none      none    none      none           
first-volume/.system          none      none    none      none         
first-volume/.system/cores    none      none    none      none         
first-volume/.system/samba4  none      none    none      none           
first-volume/.system/syslog  none      none    none      none                   
first-volume/first-dataset    none      none    none      none                   
first-volume/main-dataset    none      none    none      none             
[root@freenas ~]#                                               

After removing the quotas and reservations, the system is now happily creating snapshots.

 

[SweetAndLow]

Huh? You talk about filling a bug then you say after you remove the quota and reservation everything works. You contradicted yourself. If it's working just let it be.

 

[human]

Hi SweetAndLow,
I see you mean, yes the snapshots are working, however, when the quotas and reservations are enabled they seem to disable the snapshot function, so that's really more my concern at this point. I want to figure out if I just configured something wrong, or if there actually is some bug in the system where the quotas and reservations interfere with the snapshot functionality. I'll try to play around with different configurations of quota and reservation settings to see if I can reverse engineer what the issue is. I'll update the thread here if I figure anything out.

 

[human]

I guess this thread should actually be in bug reporting. If an admin wants to move it that would be fine...