SOLVED Unable can't take snapshot- not enough space error

[n8lbv]

Short version:

1- I'm trying to copy a zvol used for iscsi over to a new dataset.
2- Reading suggests I need to take a snapshot first and actually copy that over using zfs send.
3- I'm stuck not able to take a snapshot (not enough space error).
4- Probably due to my misprovisoning using/allocating too much diskspace for the dataset.
5- Looking to figure out if there really is enough free space to make this work and make it work
without wiping out the source dataset/zvol. or force downsizing it.
of course I'm not sure how to force downsize it either and the gui does not allow this for obvious reasons.



----

I'm very new to FreeNAS and am trying to get my head around why I can't take my first snapshot.
This is just a test system that I am trying to learn on.
This is a zvol being used for ISCSI.
It's the first and only snapshot I'm trying to take.
And yes I'm sure it's because "I'm out of space" like it reports.
And something to do with how I misprovisioned the system.
I've been reading and reading and have more reading to go obviously.

The situation I have created is that I have the working zvol/iscsi on a two (two 2TB) drive mirror, intentionally failed one drive
and is running and working on one drive. (degraded state)
That should have nothing to do with what I am trying to do but is mentioned in case it comes up later for any reason.
Yes I can simply install a new drive and rebuild but that is NOT what I am trying to teach/put myself through here.
-Sorry sidetrack-
I have added a new dataset which is a 2TB stripe of two 1TB drives.

The task I am trying to complete (and learn from) is to simply copy the existing zvol over to the newly created dataset.
Really that's it.
For this test it really doesn't matter that it's two 1TB drives in a stripe, it could just be a new 2TB or 4 TB drive
That the new dataset is on for all I care.

From what I'm reading I'm supposed to create a snapshot and basically use zfs send to get a copy over to the new dataset.
But I am running into the out of space error..

From the ISCSI view there is 640GB of 1600GB free (see my misprovisioning?) I used way more than 50% of what was available.
Freenas shows 1.6GB used and 134GB available..
Just from those numbers and no experience I'd think my one and only snapshot should work.

Been at it a little long.
Trying to get my head around *this*
https://docs.oracle.com/cd/E19253-01/819-5461/gbdbb/index.html

And trying to determine if once I figure this out if it is what I need to adjust
in or der to get the snapshot to fly before I try to send it over to the new dataset.
Am I even barking up the right tree here?

The end goal is that I just want to see if I can actually copy a test zvol used for iscsi over to a new fresh dataset.

Thanks for pointers and patience.



zfs list
NAME USED AVAIL REFER MOUNTPOINT
Stripey 544K 1.76T 88K /mnt/Stripey
Twin-2 1.63T 134G 88K /mnt/Twin-2
Twin-2/.system 9.77M 134G 96K legacy
Twin-2/.system/configs-9a9c823ea6d045b2b2330e3252de4f8d 200K 134G 200K legacy
Twin-2/.system/cores 644K 134G 644K legacy
Twin-2/.system/rrd-9a9c823ea6d045b2b2330e3252de4f8d 8.25M 134G 8.25M legacy
Twin-2/.system/samba4 268K 134G 268K legacy
Twin-2/.system/syslog-9a9c823ea6d045b2b2330e3252de4f8d 260K 134G 260K legacy
Twin-2/.system/webui 88K 134G 88K legacy
Twin-2/ZV1 1.63T 1016G 782G -
freenas-boot 762M 6.04G 64K none
freenas-boot/ROOT 761M 6.04G 29K none
freenas-boot/ROOT/Initial-Install 1K 6.04G 756M legacy
freenas-boot/ROOT/default 761M 6.04G 756M legacy

 

[n8lbv]

I'm not just sitting here waiting for an answer.
Took a nap and a shower and now am putting myself through a ZFS primer.
Had to happen sooner or later.
Seems that at the command prompt I'm able to take a snapshot without any errors.
And it creates it without any errors on the console anyhow.
I have no clue if it's a real functional snapshot or not yet still learning.
It's a zvol so I can't just go in there and verify files or anything like that that I've done before.

I have to now learn how to handle a zvol snapshot, figure out if it's real and how the heck to copy it over to a new dataset!

 

[n8lbv]

I'm still in the dark and partially clueless.
I still don't know of my snapshot is "real".


Tried :

earlier when creating the snapshot:
with no errors on the console-

Now tried:

zfs send Twin-2@testShot | zfs recv -F stripe

Comes back to prompt with no errors and also no hard drive activity or change in destination dataset size over time
So it's not doing anything.


Well crap-a-roni

My snapshot *earlier* must have not been recursive.
I still have the out of space issue to figure out.



root@FN2[/mnt]# zfs snapshot -r Twin-2@snap2
cannot create snapshot 'Twin-2/ZV1@snap2': out of space
no snapshots were created

 

[n8lbv]

I think it would have worked fine had I thin provisioned it to start out with.
But now I don't think it can be changed.

There is the slow boat to China method and that that is to mount the iscsi share in Windows and mount another new one and
copy the 777GB of data over manually in Windows to a new properly provisioned zvol/iscsi share

But this is not real customer data and I want to learn something and how to do it with zfs in FreeNAS.

 

[n8lbv]

If I could just resize that down a little bit.
I'm not sure if that's the route I should look into-
Or if I need to followup on the reservations/refreservation stuff
that I definitely don't understand yet.
Back to zfs training...

 

[n8lbv]

Mod note: this message was moved here from its own thread and is a sort of tl;dr of what's going on.

This is just a test system.
I am trying to learn how to snapshot a zvol then
copy (zfs send | receive it to a new pool).
But I can't get a snapshot due to "out of space" error.
It "looks" like I have plenty of space.
My limited understanding is that a snapshot (the one and only snapshot in this case if I can get one) is a reference
to the data and should not take up much space.
Any pointers to help me understand what's going on with a zvol snapshot (does it actually need a lot of space?) and what I need to do to actually get a snapshot of it that I can send a copy over to a new pool would be greatly appreciated.
I have another thread open on this as well at: https://www.ixsystems.com/community/threads/unable-can't-take-snapshot-not-enough-space-error.77675/

Thought maybe I should ask again in shorter format.

Thanks!

Code:

root@FN2[/mnt]# zfs snapshot -r Twin-2@snapshot
cannot create snapshot 'Twin-2/ZV1@snapshot': out of space
no snapshots were created
root@FN2[/mnt]# zfs snapshot -r Twin-2/ZV1@snapshot
cannot create snapshot 'Twin-2/ZV1@snapshot': out of space
no snapshots were created
root@FN2[/mnt]# zfs list
NAME                                                      USED  AVAIL  REFER  MOUNTPOINT
New-Pool                                                  704K  1.76T    88K  /mnt/New-Pool
New-Pool/Test-zvol                                         88K  1.76T    88K  /mnt/New-Pool/Test-zvol
Twin-2                                                   1.39T   371G    88K  /mnt/Twin-2
Twin-2/.system                                           9.88M   371G    96K  legacy
Twin-2/.system/configs-9a9c823ea6d045b2b2330e3252de4f8d   312K   371G   312K  legacy
Twin-2/.system/cores                                      600K   371G   600K  legacy
Twin-2/.system/rrd-9a9c823ea6d045b2b2330e3252de4f8d      8.25M   371G  8.25M  legacy
Twin-2/.system/samba4                                     276K   371G   276K  legacy
Twin-2/.system/syslog-9a9c823ea6d045b2b2330e3252de4f8d    300K   371G   300K  legacy
Twin-2/.system/webui                                       88K   371G    88K  legacy
Twin-2/ZV1                                               1.39T  1016G   782G  -
freenas-boot                                              762M  6.04G    64K  none
freenas-boot/ROOT                                         761M  6.04G    29K  none
freenas-boot/ROOT/Initial-Install                           1K  6.04G   756M  legacy
freenas-boot/ROOT/default                                 761M  6.04G   756M  legacy
root@FN2[/mnt]#

 

[Ericloewe]

Please use CODE tags when pasting console output, otherwise it is often unreadable. I had to edit them into your post just to figure out what was going on.

What's the output of zfs list -r -t space Twin-2?

 

[n8lbv]

Will do!
Thanks for the thread move as well.
-
I'm not sure what to do with the -t
>>invalid type 'space'

Code:

root@FN2[~]# zfs list -r Twin-2
NAME                                                      USED  AVAIL  REFER  MOUNTPOINT
Twin-2                                                   1.39T   371G    88K  /mnt/Twin-2
Twin-2/.system                                           9.15M   371G    96K  legacy
Twin-2/.system/configs-9a9c823ea6d045b2b2330e3252de4f8d   312K   371G   312K  legacy
Twin-2/.system/cores                                      548K   371G   548K  legacy
Twin-2/.system/rrd-9a9c823ea6d045b2b2330e3252de4f8d      7.55M   371G  7.55M  legacy
Twin-2/.system/samba4                                     284K   371G   284K  legacy
Twin-2/.system/syslog-9a9c823ea6d045b2b2330e3252de4f8d    308K   371G   308K  legacy
Twin-2/.system/webui                                       88K   371G    88K  legacy
Twin-2/ZV1                                               1.39T  1016G   782G  -
root@FN2[~]#



------------------------------------------------------

root@FN2[~]# zfs list -r -t space Twin-2
invalid type 'space'
usage:
        list [-Hp] [-r|-d max] [-o property[,...]] [-s property]...
            [-S property]... [-t type[,...]] [filesystem|volume|snapshot] ...

The following properties are supported:

        PROPERTY       EDIT  INHERIT   VALUES

        available        NO       NO   <size>
        clones           NO       NO   <dataset>[,...]
        compressratio    NO       NO   <1.00x or higher if compressed>
        creation         NO       NO   <date>
        defer_destroy    NO       NO   yes | no
        logicalreferenced  NO       NO   <size>
        logicalused      NO       NO   <size>
        mounted          NO       NO   yes | no
        origin           NO       NO   <snapshot>
        receive_resume_token  NO       NO   <string token>
        refcompressratio  NO       NO   <1.00x or higher if compressed>
        referenced       NO       NO   <size>
        type             NO       NO   filesystem | volume | snapshot | bookmark
        used             NO       NO   <size>
        usedbychildren   NO       NO   <size>
        usedbydataset    NO       NO   <size>
        usedbyrefreservation  NO       NO   <size>
        usedbysnapshots  NO       NO   <size>
        userrefs         NO       NO   <count>
        written          NO       NO   <size>
        aclinherit      YES      YES   discard | noallow | restricted | passthrough | passthrough-x
        aclmode         YES      YES   discard | groupmask | passthrough | restricted
        atime           YES      YES   on | off
        canmount        YES       NO   on | off | noauto
        casesensitivity  NO      YES   sensitive | insensitive | mixed
        checksum        YES      YES   on | off | fletcher2 | fletcher4 | sha256 | sha512 | skein
        compression     YES      YES   on | off | lzjb | gzip | gzip-[1-9] | zle | lz4
        copies          YES      YES   1 | 2 | 3
        dedup           YES      YES   on | off | verify | sha256[,verify], sha512[,verify], skein[,verify]
        devices         YES      YES   on | off
        exec            YES      YES   on | off
        filesystem_count YES       NO   <count>
        filesystem_limit YES       NO   <count> | none
        jailed          YES      YES   on | off
        logbias         YES      YES   latency | throughput
        mlslabel        YES      YES   <sensitivity label>
        mountpoint      YES      YES   <path> | legacy | none
        nbmand          YES      YES   on | off
        normalization    NO      YES   none | formC | formD | formKC | formKD
        primarycache    YES      YES   all | none | metadata
        quota           YES       NO   <size> | none
        readonly        YES      YES   on | off
        recordsize      YES      YES   512 to 1M, power of 2
        redundant_metadata YES      YES   all | most
        refquota        YES       NO   <size> | none
        refreservation  YES       NO   <size> | none
        reservation     YES       NO   <size> | none
        secondarycache  YES      YES   all | none | metadata
        setuid          YES      YES   on | off
        sharenfs        YES      YES   on | off | share(1M) options
        sharesmb        YES      YES   on | off | sharemgr(1M) options
        snapdir         YES      YES   hidden | visible
        snapshot_count  YES       NO   <count>
        snapshot_limit  YES       NO   <count> | none
        sync            YES      YES   standard | always | disabled
        utf8only         NO      YES   on | off
        version         YES       NO   1 | 2 | 3 | 4 | 5 | current
        volblocksize     NO      YES   512 to 128k, power of 2
        volmode         YES      YES   default | geom | dev | none
        volsize         YES       NO   <size>
        vscan           YES      YES   on | off
        xattr           YES      YES   on | off
        userused@...     NO       NO   <size>
        groupused@...    NO       NO   <size>
        userquota@...   YES       NO   <size> | none
        groupquota@...  YES       NO   <size> | none
        written@<snap>   NO       NO   <size>

Sizes are specified in bytes with standard units such as K, M, G, etc.

User-defined properties can be specified by using a name containing a colon (:).

The {user|group}{used|quota}@ properties must be appended with
a user or group specifier of one of these forms:
    POSIX name      (eg: "matt")
    POSIX id        (eg: "126829")
    SMB name@domain (eg: "matt@sun")
    SMB SID         (eg: "S-1-234-567-89")
root@FN2[~]#

 

[Ericloewe]

Sorry, it's -o space.

 

[n8lbv]

Thanks!

Code:

root@FN2[~]# zfs list -r -o space Twin-2
NAME                                                     AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
Twin-2                                                    371G  1.39T         0     88K              0      1.39T
Twin-2/.system                                            371G  9.19M         0     96K              0      9.10M
Twin-2/.system/configs-9a9c823ea6d045b2b2330e3252de4f8d   371G   312K         0    312K              0          0
Twin-2/.system/cores                                      371G   548K         0    548K              0          0
Twin-2/.system/rrd-9a9c823ea6d045b2b2330e3252de4f8d       371G  7.59M         0   7.59M              0          0
Twin-2/.system/samba4                                     371G   284K         0    284K              0          0
Twin-2/.system/syslog-9a9c823ea6d045b2b2330e3252de4f8d    371G   308K         0    308K              0          0
Twin-2/.system/webui                                      371G    88K         0     88K              0          0
Twin-2/ZV1                                               1016G  1.39T         0    782G           645G          0
root@FN2[~]#

 

[Ericloewe]

It shouldn't work, but try getting rid of that reservation on the zvol. There are possibly some weird edge case bugs here (some of them possibly fixed already in the "new" ZoL-derived code that's coming soonish to FreeNAS), so let's start with that.

 

[n8lbv]

Will give that a try.
At the moment I'm not familiar with reservations or how to remove them.
Reading/learning now and will be back soon.

 

[n8lbv]

Still trying.
I don't know how to remove a reservation.
I keep finding posts like this one https://www.ixsystems.com/community/threads/cannot-create-snapshot-out-of-space.6957/

And it always goes like this:

" I can't tell why then... I would try to remove the refreservation just to make sure it is unrelated, even though you have enough free space."

" Good advice, I removed the refreservations and now I'm able to create new datasets and snapshots. Must be a bug somewhere... anyway, thank you for your time!"

Back to reading!

 

[n8lbv]

Oh figured it out...

zfs set refreservation=none

Did not find that anywhere... but figured it out based on some examples found here: https://docs.oracle.com/cd/E19253-01/819-5461/gbdbb/index.html
And the readback seen from other datasets without a reservation.

Thought I'd try "none" as per readback on other datasets and whamo!
Will try the snapshot again,

 

[n8lbv]

Snapshot succeeded.
I feel like a real man again.
Now to break something and just send it.

 

[n8lbv]

Full Send in progress!

 

[Ericloewe]

Figures. This will have to be looked into again after the new code is in, too see if the problem persists.

 

[n8lbv]

So I'm at the next problem now.
I've copied the data to the new pool it went in a couple of hours and with no errors.
I can see it's now taking up the expected amount space on the new pool : (New-Pool)

Code:

root@FN2[~]# zfs list
NAME                                                      USED  AVAIL  REFER  MOUNTPOINT
New-Pool                                                  895G   903G   895G  -
Twin-2                                                    782G  1016G    88K  /mnt/Twin-2
Twin-2/.system                                           10.4M  1016G    96K  legacy
Twin-2/.system/configs-9a9c823ea6d045b2b2330e3252de4f8d   424K  1016G   424K  legacy
Twin-2/.system/cores                                      548K  1016G   548K  legacy
Twin-2/.system/rrd-9a9c823ea6d045b2b2330e3252de4f8d      8.64M  1016G  8.64M  legacy
Twin-2/.system/samba4                                     284K  1016G   284K  legacy
Twin-2/.system/syslog-9a9c823ea6d045b2b2330e3252de4f8d    332K  1016G   332K  legacy
Twin-2/.system/webui                                       88K  1016G    88K  legacy
Twin-2/ZV1                                                782G  1016G   782G  -
freenas-boot                                              762M  6.04G    64K  none
freenas-boot/ROOT                                         761M  6.04G    29K  none
freenas-boot/ROOT/Initial-Install                           1K  6.04G   756M  legacy
freenas-boot/ROOT/default                                 761M  6.04G   756M  legacy

However the data is not in the correct format and/or structure and ready to use.
If I try to associate an iscsi share with the "New-Pool" it does not show up as an option.

Looks like I messed up somewhere and need to try over.
Looking in the GUI

My original working Pool (Twin-2) at the top(root) type is "dataset"
and then there's a zvol (type "zvol" named ZV1 under it)


My new pool (New-Pool) is type zvol at the top with nothing under it.

I'm not sure if this is something I did wrong/different when I created it, or if this is the result
after I zfs sent the blob of data over to the new pool.

I'm going to create an additional new pool and trace my steps as I go and keep an eye on that structure.

 

[n8lbv]

When creating a new pool from a new device it definitely creates type dataset at the top (root).
What I'm trying this time is I created a new zvol under the new pool and this time I am zfs sending to that zvol under the dataset.
In a few hours I'll be able to test it.

 

[n8lbv]

Oh man that worked!
Everything zfs all is fresh new and exciting to me.
After copying the snapshot it was still a 'um snapshot!
I had to "rollback" to make it read/write.
ISCSI shared it. ( I can use caps for ISCSI now that it works).
It works and is on a new drive set like I was after!