Lost files/disk size

[Peter G]

Hi,

Running FreeNAS-9.1.0-RELEASE-x64 (dff7d13) ...
AMD A8-5600K APU with Radeon(tm) HD Graphics
15818MB
6 disks...

My problem:
I have a mirrored encrypted ZFS volume with have CIFS share or I should HAD. (2 disk a 500 GB)

Yesterday I tested copying a large amount of file to the share . I was using a Linux Ubuntu 12.04 and I wanted to see how it handled disk near full conditions.

So I arranged two a almost similar sources on the Ubuntu and copied them one after the other to the FreeNAS. On start of the second copying run it warned, as it should, that the volume was not big enough, but I pressed Proced anyway... then I answered that it should skip coping identical files...

Time passes and the you hit disk full... abort the copy process on the Ubuntu machine... browsed the files... seemed to be there... made a reboot ... and mounted the encrypted disks again.

What I found was one ZFS pool with a free space of 800 kbytes (less than the size of the file that did not fit on the disk during copy) and a total size of the disk is now set to 900 kbytes...and NO files. Before this operations it the size of the ZFS pool about 485 GB...

Have I lost the files? Resilvering/scrubbing does not do anyting... it only resilvers/scrubs the "900kbyte" disk it seems...

Any ideas on how to restore the volume to it proper size without losing data on it?

Is this a bug I have stumbled upon?

Best regards
Peter G

 

[cyberjock]

Post the output of zpool status and zpool import, and put it in CODE as the formatting of the text matters.

 

[Peter G]

Thx for the help... regretfully I will not be able to post the requested data until Wednesday (traveling) but will post ASAP. :)

 

[Peter G]

Hi again,
Note correctly formated text in attached Report.txt

Here is the result of zpool status

Code:

 
pool: Safe state: ONLINE scan: scrub in progress since Sun Aug 25 14:23:43 2013 261G scanned out of 453G at 49.8M/s, 1h5m to go 0 repaired, 57.66% done config: NAME STATE READ WRITE CKSUM Safe ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gptid/39ffe0b5-00d3-11e3-adb0-bc5ff4a2130c.eli ONLINE 0 0 0 gptid/3af4165b-00d3-11e3-adb0-bc5ff4a2130c.eli ONLINE 0 0 0 errors: No known data errors
and zpool import:
[root@BIGDATA ~]# zpool import
[root@BIGDATA ~]#
 
Result none...
 
[root@BIGDATA ~]# zpool list                                                   
zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT Bigsafe 1.81T 291G 1.53T 15% 1.00x ONLINE /mnt Media 2.72T 2.07T 665G 76% 1.00x ONLINE /mnt Safe 460G 453G 7.19G 98% 1.00x ONLINE /mnt // note the allocated figure           
[root@BIGDATA ~]#                                     

Actice volumes on webpage says:

Safe Used 3.0KiB(3%) Available 755.0 KiB Size 832,5 KiB Status HEALTHY

Any ideas...?

Thx for your support and help...

/Peter G

 

[cyberjock]

Yeah... repost that in CODE. Without the formatting of the data it is useless.

 

[cyberjock]

Safe isn't a zpool, it's a zvol I'm guessing. Do you have snapshots enabled for the zvol or the actual pool? Something is taking up space...

That really doesn't provide the formatting either. The formatting is what matters and you "flattened" it. But it doesn't matter, I got enough from what you've posted.

 

[Peter G]

Safe isn't a zpool, it's a zvol I'm guessing. Do you have snapshots enabled for the zvol or the actual pool? Something is taking up space...

That really doesn't provide the formatting either. The formatting is what matters and you "flattened" it. But it doesn't matter, I got enough from what you've posted.

Sorry for the formatting, doing it remotely only by webinterface...and a crappy lent computer.

No known snapshots... as the failure happend just after disk was filled.

 

[Peter G]

Checking my documentation... There was a zvol Safe and on it there was a dataset Data... which was a CIFS share (the one that was completly filled). It is the one missing and would account for the data amount allocated. The disks are mirrored and encrypted... decrypt ok no errors.

Guess something went very wrong... no data lost as I have backups...but I would like to know what happend. (and you too I suppose).

thx for your time and patience

/Peter G

 

[cyberjock]

Well, something is taking up the space. This is a common issue as people don't recognize what is taking up space where. Most commonly people don't understand what a given app uses to calculate used and free space.

Here's 2 different commands that give 2 totally different results.. so which is correct?

Code:

[root@freenas] ~# zpool list
NAME  SIZE  ALLOC  FREE    CAP  DEDUP  HEALTH  ALTROOT
tank  32.5T  29.9T  2.57T    92%  1.00x  DEGRADED  /mnt
[root@freenas] ~# df -h
Filesystem            Size    Used  Avail Capacity  Mounted on
/dev/ufs/FreeNASs2a    926M    567M    285M    67%    /
devfs                  1.0k    1.0k      0B  100%    /dev
/dev/md0              4.6M    3.8M    387k    91%    /etc
/dev/md1              823k    2.0k    756k    0%    /mnt
/dev/md2              149M    19M    118M    14%    /var
/dev/ufs/FreeNASs4      19M      2M    16M    11%    /data
tank                    23T    21T    1.5T    94%    /mnt/tank

So does my zpool "tank" have 2.57T free or 1.5TB free?

FACT: I actually have 1.5TB available to use as I please.

FACT: If you look at the output of df and zpool list you'll see that the total size and allocated don't match either.

Are you confused yet? Me too.

 

[Peter G]

Agree, i am confused too... as there is the third "figure" on the webpage in my case where I have

Safe Used 3.0KiB(3%) Available 755.0 KiB Size 832,5 KiB Status HEALTHY

and if i try any operation like create a new dataset S ... it fails with error:

cannot create 'Safe/s': out of space

Starting think I lost something more ... like sanity.
/Peter

 

[cyberjock]

nope.. you just need to go and read up on what the programs use to get their values and figure out where the "missing" data is. I'm sure your data is safe, you just don't have the requisite knowledge to understand what is going on.

 

[Peter G]

Hum, now I am really confused...

Made the following:

Rebooted
Unlocked the zvol Safe (with passphrase, ok)
Still no data visible and size of disk 823,5 Kib
zpool export Safe (as it said there is one)
zpool import Safe Safe2

!!! got error unable the get disk size if I try to unlock Safe in webinterface
so I
zpool import -R /mnt Safe2 Safe

and
dataset Data is back with 465 GByte (100% full)

So YES I am confused...
It looks like the mount point has been "overwriten/forgotten/mashed" by the filling of the disk?

Hope someone check this out...this is beyond my knowledge and understanding.

Probably a case of RTFM and SBK ...I rest my case ...
/Peter G

 

[cyberjock]

Um.. how is your zvol encrypted? FreeNAS' built-in encryption is only officially supported for zpools.

 

[Peter G]

Sorry for the confusion... an extra z dropped in... you are completely right.
I just discovered ...Re-read the manual and found the term zpol as a block device... (so I understand your confusion).

I also noticed that naming conventions are playing havoc with our conversation... (and you being an advanced user and me being new to this platform)
WebGUI named ZFS Volume is a zpool in shell (Correct ?)
thus my added confusion...and yours...

I thus have a encrypted zpool (ZFS Volume) Safe with an CIFS share Safe and on the zpool (ZFS Volume) Safe there is an Dataset (webGUI lingo...not worked with these in shell yet)Data.

As I stated earlier I got the data visible again but it only lasts the session until reboot. Will be back later today at the physical location and
will then try to remove some files to make room and see if it is lack of space on the zpool (ZFS Volume) that is causing the inability to mount the zpool (ZFS Volume) properly.

Thx for your patience with me and my problem.

 

[gpsguy]

If you discover that you can't delete files, due to the disk full condition, read this thread for a solution.

 

[Peter G]

If you discover that you can't delete files, due to the disk full condition, read this thread for a solution.

Thx for the tip... I think the filesystem is full ...and yes it will kick back any attempts to delete... but a simple null pipe... yes, can work.

Lesson learned... Set Quota to make sure you never reach 100%... 90% perhaps... then you can lift Quota and recover... then set it back (Quota)...(perhaps... will test)
/Peter G

The END... made the export/import ... got data visible...could not delete... used above tip... and got 2 files off the disk... now it mounts properly after reboot.

Lesson learned 2: Always have an offline copy on another machine and location (at least).

Thx all for helping me out.