[Errno 28] No space left on device

[JohnGoutbeck]

Hello All;

had to shutdown the FreeNAS 9.3 storage. Of course I did not take a backup first and of the FreeNAS configuration. It did not come backup properly. Seems to have lost its mind.

So on an another 8GB USB stick I installed a fresh copy of FreeNAS 9.3 from the March release and recovered from the last config and rebooted. the system will state that there is no space left on the device.

Downloaded and installed a 9.3.1 (September release). If I do a volume import I will get this;

----
Import Volume

Request Method: POST
Request URL: http://192.168.12.233/storage/auto-import/
Software Version: FreeNAS-9.3-STABLE-201509022158
Exception Type: OSError
Exception Value:

[Errno 28] No space left on device: '/var/db/system/cores'

Exception Location: /usr/local/www/freenasUI/../freenasUI/middleware/notifier.py in system_dataset_create, line 5450
Server time: Tue, 24 Nov 2015 13:18:43 -0800

--

Environment:

Software Version: FreeNAS-9.3-STABLE-201509022158
Request Method: POST
Request URL: http://192.168.12.233/storage/auto-import/


Traceback:
File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
105. response = middleware_method(request, callback, callback_args, callback_kwargs)
File "/usr/local/www/freenasUI/../freenasUI/freeadmin/middleware.py" in process_view
157. return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
22. return view_func(request, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py" in view
69. return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/django/contrib/formtools/wizard/views.py" in dispatch
236. response = super(WizardView, self).dispatch(request, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py" in dispatch
87. return handler(request, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/django/contrib/formtools/wizard/views.py" in post
297. return self.render_done(form, **kwargs)
File "/usr/local/lib/python2.7/site-packages/django/contrib/formtools/wizard/views.py" in render_done
350. done_response = self.done(final_form_list, **kwargs)
File "/usr/local/www/freenasUI/../freenasUI/storage/forms.py" in done
947. _n.restart("system_datasets")
File "/usr/local/www/freenasUI/../freenasUI/middleware/notifier.py" in restart
368. self._simplecmd("restart", what)
File "/usr/local/www/freenasUI/../freenasUI/middleware/notifier.py" in _simplecmd
243. f()
File "/usr/local/www/freenasUI/../freenasUI/middleware/notifier.py" in _restart_system_datasets
5226. systemdataset = self.system_dataset_create()
File "/usr/local/www/freenasUI/../freenasUI/middleware/notifier.py" in system_dataset_create
5450. os.chmod(corepath, 0775)

Exception Type: OSError at /storage/auto-import/
Exception Value: [Errno 28] No space left on device: '/var/db/system/cores'
---
I get this same error for the two imported pools, bu the pools, datasets and volumes show up under Storage
I do not access to the real data
--
[root@freenas /mnt/V1/.freenas]# zfs list
NAME USED AVAIL REFER MOUNTPOINT
V1 913G 0 137M /mnt/V1
V1/.system 1.30M 0 160K legacy
V1/.system/cores 144K 0 144K legacy
V1/.system/rrd-45c1bed4ebc7401eab348dfdc227be73 144K 0 144K legacy
V1/.system/samba4 196K 0 196K legacy
V1/.system/syslog-45c1bed4ebc7401eab348dfdc227be73 692K 0 692K legacy
V1/sbd 10.6M 10.4M 120K -
V1/z1 913G 878G 35.0G -
V2 2.63T 103M 96K /mnt/V2
V2/.system 964K 103M 464K /mnt/V2/.system
V2/.system/cores 96K 103M 96K /mnt/V2/.system/cores
V2/.system/rrd-bd9055260d994501a6a389f2eacb609a 96K 103M 96K /mnt/V2/.system/rrd-bd9055260d994501a6a389f2eacb609a
V2/.system/samba4 104K 103M 104K /mnt/V2/.system/samba4
V2/.system/syslog-bd9055260d994501a6a389f2eacb609a 204K 103M 204K /mnt/V2/.system/syslog-bd9055260d994501a6a389f2eacb609a
V2/z2 2.63T 2.53T 105G -
freenas-boot 520M 6.52G 31K none
freenas-boot/ROOT 513M 6.52G 25K none
freenas-boot/ROOT/Initial-Install 1K 6.52G 510M legacy
freenas-boot/ROOT/default 513M 6.52G 511M legacy
freenas-boot/grub 6.79M 6.52G 6.79M legacy
---
[root@freenas /mnt/V1/.freenas]# cd /mnt/V1/z1
bash: cd: /mnt/V1/z1: No such file or directory
[root@freenas /mnt/V1/.freenas]# cd /mnt/V2/z2
bash: cd: /mnt/V2/z2: No such file or directory
----

What would be the next step to recovery.?

 

[cyberjock]

The zpool with your .system dataset has 0 bytes free. The system cannot modify and log its services properly, so the system is totally borked. So how long did you ignore that warning in the WebGUI that says "don't fill the zpool more than 80% full"? :P

 

[JohnGoutbeck]

Apparently I did not notice for a time.

I'm sure I had under 80% used space (over 20% free) before the shutdown, from a remote server iSCSI LVM mount view. Did not check from the FreeNAS/zpool view.

Anyways, is there a way to mount the zpool just to clean up some files? Even from a FreeBSD point?

I really need to get at the files.

Any suggestions, helpful comments?

Have a good day.

 

[JohnGoutbeck]

OK. I got the system working without loosing data!

Followed the doc on how to expand the volume/zpool by replacing the HDs one at a time, letting the system resliver. I did this for both volumes and now I the system can reboot without errors. I then setup the iSCSI targets and extents, started the iSCSI initiators on the servers, and the server sees all the LUNS and can mount the LVMs and well as the all the data. Now I have more free space.

---
How about a zfs recovery enhancement!

Since the zfs system saves writes to the drives as transactions, it would be great it the developers could create an app or function to read the last transactions and allow the sysadmin to undo or delete the transaction hopefully freeing up some space to mount the volume/zpool.

 

[devnullius]

How about a zfs recovery enhancement!

Sorry for bumping but I just *HAD* to second that...! :)