SOLVED NFS volume offline need help.

[aim9xray]

Hi

I'm running FreeNAS-8.3.1-RELEASE-x64 (r13452) on a Dell PE2900 (Intel(R) Xeon(R) CPU 5130 @ 2.00GHz) with 16363MB of memory, that is presenting NFS shares to two VMware ESXis.
I don't exactly know what has happened, but at one point in time the ESXis lost connection to the FreeNAS box and I decided to reboot it.
After the reboot, i got that blinking yellow light in the top right corner of ther web interface saying" WARNING: The volume PE2900 (ZFS) status is UNKNOWN"
Under the volumes tab it shows "PE2900 /mnt/PE2900 0 (Error) Error getting available space Error getting total space UNKNOWN"
If i click any of the icons in "available options" I get this:

Code:

Request Method:	GET
Request URL:	http://10.15.4.230/storage/volume/zfs-edit/1/
Software Version:	FreeNAS-8.3.1-RELEASE-x64 (r13452)
Exception Type:	KeyError
Exception Value:	
'compression'
Exception Location:	/usr/local/www/freenasUI/../freenasUI/storage/forms.py in __init__, line 1094
Server time:	Tue, 21 May 2013 10:46:12 -0500


Traceback
Environment:

Software Version: FreeNAS-8.3.1-RELEASE-x64 (r13452)
Request Method: GET
Request URL: http://10.15.4.230/storage/volume/zfs-edit/1/


Traceback:
File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  105.                         response = middleware_method(request, callback, callback_args, callback_kwargs)
File "/usr/local/www/freenasUI/../freenasUI/freeadmin/middleware.py" in process_view
  166.         return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  20.                 return view_func(request, *args, **kwargs)
File "/usr/local/www/freenasUI/../freenasUI/storage/views.py" in zfsvolume_edit
  513.     volume_form = forms.ZFSVolume_EditForm(mp=mp)
File "/usr/local/www/freenasUI/../freenasUI/storage/forms.py" in __init__
  1094.         self.fields['volume_compression'].initial = data['compression']

Exception Type: KeyError at /storage/volume/zfs-edit/1/
Exception Value: 'compression'

This is an emergency for me, any help would be greatly appreciated

 

[cyberjock]

Re: NFS volume offline need help.(EMERGENCY)

WHen you say you decided to reboot.. are you talking the ESXi box or the FreeNAS box?

Can you post the output of the following commands in CODE tags?

zpool import

zpool status

gpart list

And do yourself a favor and don't trying doing stuff. If you do you may make things worse...

 

[aim9xray]

Re: NFS volume offline need help.(EMERGENCY)

WHen you say you decided to reboot.. are you talking the ESXi box or the FreeNAS box?

Can you post the output of the following commands in CODE tags?

zpool import

zpool status

gpart list

And do yourself a favor and don't trying doing stuff. If you do you may make things worse...

Hi cyberjock and thanks for your quick reply.

The buddy of mine who initially set up the box has moved to Europe and isn't available to help, so I really appreciate your effort (and no, I won't try doing stuff before I exactly know what i'm doing)!!

I had rebooted the FreeNAS box remotely but it didn't help at all and now I know way:
I drove to the colo-datacenter where the box is hosted last afternoon, to find that one of the disks had failed and the RAID volume (8 Disks on a PERC4 in RAID-0 config) is offline.

I was under the assumption that ZFS does some striping, so now I really don't know how to proceed. Do you think the data will be salvageable?

 

[cyberjock]

Re: NFS volume offline need help.(EMERGENCY)

As a RAID0, you have lost data unless you can get your drive back online. RAID0 is the most dangerous because you have no protection from a failed(or failing) disk.

If you get another disk you could try doing a ddrescue to the new disk and hope it works. After that you'd have to do a scrub and hope not too many files are corrupted.

If the drive isn't even detected by your BIOS or you can't otherwise run a ddrescue your only other option is to mail the drive to one of those data recovery companies and for 4-figures they'll try to recover the data on that disk.

A RAID0 of 8 drives was "data-suicide".

 

[aim9xray]

Re: NFS volume offline need help.(EMERGENCY)

As a RAID0, you have lost data unless you can get your drive back online. RAID0 is the most dangerous because you have no protection from a failed(or failing) disk.

If you get another disk you could try doing a ddrescue to the new disk and hope it works. After that you'd have to do a scrub and hope not too many files are corrupted.

If the drive isn't even detected by your BIOS or you can't otherwise run a ddrescue your only other option is to mail the drive to one of those data recovery companies and for 4-figures they'll try to recover the data on that disk.

A RAID0 of 8 drives was "data-suicide".

Yeah well... it's "only" 8 TB
I guess he did it because everything he (we) read about FreeNAS filers basically said "do a RAID-0 and let FreeNAS do the striping".
Since I have to rebuild it anyway now, I might as well do it from scratch and learn something.
Can you point me towards a good step by step howto or "best practices" on building a FreeNAS filer?

 

[titan_rw]

Re: NFS volume offline need help.(EMERGENCY)

FreeNAS filers basically said "do a RAID-0 and let FreeNAS do the striping".

HUH?

Freenas will absolutely give you redundancy. If you configure it to give you redundancy. I'm still shaking my head. That's like saying "my $1500 raid card will give me redundancy if I use raid 0. It's a raid card, so I'll let it do it." Freenas / zfs wants to be your 'raid'. But you gave freenas a 'single' hard drive, the array as presented by the raid card. So freenas has no redundancy. You'd be relying on the raid card to supply redundancy. And since the underlying hardware raid array was raid 0, it has no redundancy either. I'm still amazed this was setup in production.

Can this 'raid' card be configured to IT / HBA mode? Freenas / zfs wants access to the disks directly, without any raid bios in the way. If the card can't do this, it's not the best choice for zfs.

With 8 drives, I'd setup a single raidz2 vdev. You'll have the capacity of 6 drives (minus a bit of overhead), with 2 drive redundancy. Any 2 of the 8 drives can fail, and the data should still be safe.

Please review Cyberjock's slideshow. It shows what setups have redundancy and which don't.

 

[cyberjock]

Re: NFS volume offline need help.(EMERGENCY)

Yeah well... it's "only" 8 TB
I guess he did it because everything he (we) read about FreeNAS filers basically said "do a RAID-0 and let FreeNAS do the striping".
Since I have to rebuild it anyway now, I might as well do it from scratch and learn something.
Can you point me towards a good step by step howto or "best practices" on building a FreeNAS filer?

I'm shocked nobody thought about pulling out a drive with the system on "just to see what would happen". It would have been immediately know that "OMFG this is very bad news". When I started testing my first FreeNAS server I threw some test data on it and started deliberately pulling disks just to make sure I had the redundancy I thought I had. It would have sucked horribly to thought I had 2 disk redundancy and then didn't acutually have any.