Cant import Volume, Drives look ok.

Status
Not open for further replies.

Slydogsz

Dabbler
Joined
May 7, 2016
Messages
18
I just want to say up front that any assistance is appreciated. I know everyone is volunteering their time to provide people help. Thanks for what you do, support is not always appreciated.
Thanks for your help in advance.

I admit it, I did not configure my FreeNas correctly. I ran it on older hardware, with not enough memory and no ECC memory. I didn't do a lot of things that I should of, I get that and hope to of corrected what I can.

What that doesn't change is that I have a down FreeNas. Mission critical, no but I do believe my data is still there and dont want to give up on it unless I am told it is gone. I see nothing that indicates any of the drives are bad. What I lack is the knowledge and skills to get access to the data.

Here are some details that I hope someone will be able to point me in the right direction to recover the data or tell me my drives are done for. I can accept that the data is gone but I have not seen anything that indicates there was a hardware failure. My ID10T issues are a given.


Image of crash http://imgur.com/e375vbH
camcontrol devlist http://pastebin.com/AYAj4eZ1
smartctl -x /dev/da0 http://pastebin.com/ibN9aDUR
smartctl -x /dev/da1 http://pastebin.com/ZMKRd6tK
smartctl -x /dev/da2 http://pastebin.com/5wWTjT3k
smartctl -x /dev/da3 http://pastebin.com/s4JKiB31
sysctl -a | grep mps http://pastebin.com/94bt9WgJ
sas2flash -list http://pastebin.com/LHbD4dUt

This was a 4 drive ZFS1 array with a OCZ SSD drive as cache. I have learned that the cache was a bad idea.
The array was then presented as an ISCSI Lun to a ESXi cluster which then formatted the lun as a large 7tb VMFS5 volume. More bad practices I have been told. All of this was running on a Core 2 Duo system that I had laying around with 4gb of ram. Yes I know, it should of been 8gb and ECC. Probably a under-powered CPU as well.

This system worked great for over a year, then one night I was streaming a move from my virtualized media server when the movie just stopped, I went to look at my ESXi cluster and found the crash on the FreeNas screen.

After getting chastised for not having the right hardware, I have upgraded the server to a dual quad core Xeon dell server with 16gb of ECC ram and a Dell H200 SATA/SAS controller that has been flashed to IT mode.

I removed the Volume from the config and can now boot into the FreeNas console with no issues but I cant get the Volume to Import. When I try I get lots of text scrolling by and then the server reboots. No crash logs are in /var/crash and /var/messages just show the boot messages but nothing about why the import fails.

So outside of the given "You are dumb for building your NAS this way" post that I do deserve, is there anything outside of taking this server to a data recovery specialist that I can do to get to the data or should I just give up and start over ?
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
I cant get the Volume to Import. When I try I get lots of text scrolling by and then the server reboots
As a guess, I would say maybe your pool got corrupted due to a crash caused by insufficient of RAM. If true, then probably the one chance you have is the -F option on zpool import. That's an uppercase 'F', and should initially be combined with -n for a dry run, just to see what it comes up with.
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
Also, consider removing the L2ARC device and adding the -m option to the import.
 

Slydogsz

Dabbler
Joined
May 7, 2016
Messages
18
I have ran zpool import -n -F before, I just tried it with the -m after removing the L2ARC drive. Same outcome, lots of text scrolls by and then the server reboots. Pool will mount as Read Only but I have no way of presenting the Read Only volume as an ISCSI lun and FreeBSD does not read VMFS5
 

Slydogsz

Dabbler
Joined
May 7, 2016
Messages
18
I dont have that much extra storage laying around and wont get the OK from the Mrs for the $400 it would take to purchase.
 

Slydogsz

Dabbler
Joined
May 7, 2016
Messages
18
Well I guess I am going to try and get some drives, may not have a happy wife for a while but might get my data back...What is the worst that could happen? Guess should could stop talking to me but that just means more time for Overwatch.

I can put the new pool on the same server right ?
Any more details on doing a zfs send / zfs receive ?
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
Any more details on doing a zfs send / zfs receive ?
If you follow the link to the documentation you can read all about the two commands. You would probably use the version of zfs send that works from a filesystem, unless you can identify a snapshot that's a better candidate for data recovery (hint: zfs list -t). You'll want to use the -v option to report progress, and the -n option to dry-run while you figure out exactly what you want to do. I'm assuming you know how to pipe the output of one command to the input of another - you'll be piping the output of zfs send to the input of zfs receive.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Also, consider removing the L2ARC device and adding the -m option to the import.

-m is for slog, not L2ARC.

Second, that crash indicates spacemap corruption. The only fix is to make a new zpool. BUT, you should be able to get your data off the zpool if you mount it readonly from the CLI. So you'll need to make a new zpool and then copy the data (I wouldn't try to do ZFS replication, and that's always a bad idea if zpool corruption is likely, which is virtually a certainty when dealing with kernel panics from ZFS).

Hate to say it, but this is more than likely self-induced from having your own self-identified improper hardware.

You can mount the zpool from the CLI doing something like

# zpool import -R /mnt -0 readonly=on <poolname>

Good luck.
 

philhu

Patron
Joined
May 17, 2016
Messages
258
Can I ask a small off-topic question?

Are SSD caches for ZIL and/or L2ARC a bad idea?

I have 48G system, and just added a Kingston 16g ZIL cache and a 240G L2ARC
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
Are SSD caches for ZIL and/or L2ARC a bad idea?
As usual, it depends.

A device dedicated to the ZIL is called an SLOG device, for Separate Intent Log (every pool has a ZFS Intent Log, but it's usually on the data disks). A SLOG device is only beneficial when sync writes are required and leaving the ZIL on the pool is detrimental to performance. If installed, it must have power loss protection.

An L2ARC adds a 2nd level of read caching. It consumes some RAM, which might otherwise be used for 1st level read caching. As such, it is only beneficial on a busy pool that is otherwise short of read cache capacity (large ARC, low hit ratio), and can hurt performance if the RAM it consumes would be better allocated to ARC.
I have 48G system, and just added a Kingston 16g ZIL cache and a 240G L2ARC
You probably don't need either, but you haven't described your workload, or any testing you've done to identify any performance bottlenecks.
 

philhu

Patron
Joined
May 17, 2016
Messages
258
My system is in constantly doing large file write mode transfers doing many streams of transcoding or copying 80g folders off the nas and back on it

It does seem slow with multiple transcodes running on my desktop with the nas as the disk backend using nfs

My desktop is win10 running on a i7 hex core proccessor
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
My system is in constantly doing large file write mode transfers doing many streams of transcoding or copying 80g folders off the nas and back on it

It does seem slow with multiple transcodes running on my desktop with the nas as the disk backend using nfs

My desktop is win10 running on a i7 hex core proccessor
I suggest you start your own thread and provide a lot more information.
 

philhu

Patron
Joined
May 17, 2016
Messages
258
Deal. I will tommorrow
 

Slydogsz

Dabbler
Joined
May 7, 2016
Messages
18
Second, that crash indicates spacemap corruption. The only fix is to make a new zpool. BUT, you should be able to get your data off the zpool if you mount it readonly from the CLI. So you'll need to make a new zpool and then copy the data (I wouldn't try to do ZFS replication, and that's always a bad idea if zpool corruption is likely, which is virtually a certainty when dealing with kernel panics from ZFS).

When I mount the volume as Read Only I dont see the dataset. I think this is expected because it was being presented as an ISCSI lun and it is formatted VMFS5 which BSD does not understand. Is there another way to copy the data than doing a ZFS replication ?
a ZFS list after mounting the volume read only shows this

[root@freenas] ~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
EllerData 7.22T 443G 140K /mnt/EllerData
EllerData/.system 365M 443G 330M legacy
EllerData/.system/configs-9302fb29893e4327a694006acd742b52 140K 443G 140K legacy
EllerData/.system/cores 33.3M 443G 33.3M legacy
EllerData/.system/rrd-9302fb29893e4327a694006acd742b52 140K 443G 140K legacy
EllerData/.system/samba4 186K 443G 186K legacy
EllerData/.system/syslog-9302fb29893e4327a694006acd742b52 581K 443G 581K legacy
EllerData/Digital 7.22T 4.64T 3.01T -
EllerData/jails 140K 443G 140K /mnt/EllerData/jails
freenas-boot 516M 224G 31K none
freenas-boot/ROOT 510M 224G 25K none
freenas-boot/ROOT/Initial-Install 1K 224G 496M legacy
freenas-boot/ROOT/default 510M 224G 492M legacy
freenas-boot/grub 6.33M 224G 6.33M legacy
tank 1.73M 7.65T 140K /mnt/tank
tank/.system 988K 7.65T 151K legacy
tank/.system/configs-9302fb29893e4327a694006acd742b52 140K 7.65T 140K legacy
tank/.system/cores 140K 7.65T 140K legacy
tank/.system/rrd-9302fb29893e4327a694006acd742b52 140K 7.65T 140K legacy
tank/.system/samba4 221K 7.65T 221K legacy
tank/.system/syslog-9302fb29893e4327a694006acd742b52 198K 7.65T 198K legacy

Tank is the new volume I just created with new storage that I will copy the data to, Digital is the dataset that was being shared as a ISCSI lun. It has no mount point and if I try to assign it a mount point the error message says this type of a dataset cant be assigned a mount point.

Any thoughts or ideas would be appreciated.
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
Please post the output of zfs get all EllerData/Digital between CODE tags.
code.png
 
Status
Not open for further replies.
Top