Flash drive died, reinstalled FreeNAS but now iSCSI LUN isn't mounting

[Randolfini]

Hi all, I need some help with some potential data loss. An admin prior to me setup a Windows Server 2008 R2 with iSCSI LUNS from two FreeNAS-8.3.0-RELEASE-x64 (r12701M) hosts. Dell R510, with RAID6 10.9TB LUN from each of hosts, listed as mfid0 in FreeNAS. On the file server the LUN was setup as Dynamic Disks. FreeNAS is running off a 4GB usb drive.

We lost power this past Wednesday afternoon and when power came back on one of the hosts wouldn't boot.
Reported cannot read /var/run/utmp
Warning: /data was not properly dismounted
various ufs/FreeNASs4 Read errors
md5: /data/freenas-v1.db Input/output error
Can't open /var/tmp/rc.conf.freenas: No such file or directory

I reinstalled FreeNAS-8.3.0-RELEASE-x64 (r12701M) to a new USB drive, it recognized the existing mfid0 disk. I tried to restore the config in the GUI but the config file is corrupt. When I reconfigured iSCSI and re-presented it, the Windows server only had the option to initialize the volume. It did not auto-mount it.

Help! I've already started restoring from backups to another server.

 

[HoneyBadger]

What RAID card is in the Dell R510s?

The correct answer is either "the H200, and we don't have any drives configured" or "we replaced it with an HBA" - anything else and your predecessor may have set up ZFS on hardware RAID which is a big no-no.

However,

I've already started restoring from backups to another server.

You're awesome for having backups.

Aside: If the data is already hosed and needing a restore, go to FreeNAS 9.3 as there's major improvements to iSCSI.

 

[Randolfini]

Thanks for the quick reply! My backup is 7 days old so I'm still trying to get to this data to close that gap. Naturally users are not happy about the gap either.

Raid controller is a PERC 6i details below. Prior to the crash I recall the GUI didn't list any ZFS volumes under storage. If I clicked view disks there was only mfid0.

root@rebuiltnas01 ~]# mfiutil show adapter
mfi0 Adapter:
Product Name: PERC 6/i Integrated
Serial Number: 1122334455667788
Firmware: 6.3.1-0003
RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID6, RAID10, RAID50
Battery Backup: present
NVRAM: 32K
Onboard Memory: 256M
Minimum Stripe: 8k
Maximum Stripe: 1M

[root@rebuiltnas01 ~]# mfiutil show volumes
mfi0 Volumes:
Id Size Level Stripe State Cache Name
mfid0 ( 10T) RAID-6 64k OPTIMAL Disabled <100nas01>

[root@rebuiltnas01 ~]# mfiutil show config
mfi0 Configuration: 1 arrays, 1 volumes, 0 spares
array 0 of 8 drives:
drive 0 ( 1863G) ONLINE <WDC WD2003FYYS-1 1D02 serial=WD-WCAY00245002>
SATA
drive 1 ( 1863G) ONLINE <WDC WD2003FYYS-1 1D02 serial=WD-WCAY00256315>
SATA
drive 2 ( 1863G) ONLINE <WDC WD2003FYYS-1 1D02 serial=WD-WCAY00266347>
SATA
drive 3 ( 1863G) ONLINE <WDC WD2003FYYS-1 1D02 serial=WD-WCAY00265156>
SATA
drive 4 ( 1863G) ONLINE <WDC WD2003FYYS-1 1D02 serial=WD-WCAY00218025>
SATA
drive 5 ( 1863G) ONLINE <WDC WD2003FYYS-1 1D02 serial=WD-WCAY00232647>
SATA
drive 6 ( 1863G) ONLINE <WDC WD2003FYYS-1 1D02 serial=WD-WCAY00245310>
SATA
drive 7 ( 1863G) ONLINE <WDC WD2003FYYS-1 1D02 serial=WD-WCAY00264714>
SATA
volume mfid0 (10T) RAID-6 64k OPTIMAL <100nas01> spans:
array 0

 

[HoneyBadger]

Sorry to say that your predecessor set up ZFS on top of hardware RAID. This is a 100% unsupported "do not do this EVER" configuration. Your data is most likely corrupted beyond recovery. But thankfully you have a backup.

I'd suggest you find a SAS 6i/R or PERC H200 as replacement for the PERC 6i and leave all drives unconfigured.

Edit - After further review, it looks like it might be set up to share the RAID6 array from the MegaRAID directly via iSCSI without any kind of ZFS layer in the middle.

 

[Randolfini]

: ( Well thanks for the quick responses.

I have a second FreeNAS install on another server and it booted up ok. It is in the same configuration. I'm new to FreeNAS (quick the trial by fire here), how can I see the existing ZFS configuration on it?

 

[Randolfini]

Also to clarify and make sure we are on the same page:

USB Flash drive has the FreeNAS OS installed to it.

The RAID appears healthy and not failed. When this setup was working before the crash I didn't find any reference to ZFS pools, disks, volumes, etc.

Is there anyway to confirm this is ZFS on top of hardware raid?

 

[jgreco]

Is there anyway to confirm this is ZFS on top of hardware raid?

volume mfid0 (10T) RAID-6 64k OPTIMAL <100nas01> spans:
array 0

You already confirmed it with that line right there.

 

[jgreco]

Ah, correction, I re-read this a bit. I guess there's no guarantee that it is ZFS on top of that. It confirms the low level storage is built on top of a hardware RAID6.

It's possible that the mfid0 was directly shared out as a device, in which case, I believe that was possible with FreeNAS 8 and I don't think anyone has much familiarity with it. You could probably duplicate the configurastion and it'd probably work.

 

[HoneyBadger]

@jgreco mfiutil is the FreeBSD utility for managing MegaRAIDs, if there's anything defined there it's using the hardware RAID. Edit - Wait, reread that again, if there was no pools defined then maybe no ZFS at all. It might be a directly shared device via iSCSI?

@Randolfini run "zpool status" from a shell. If it's directly on the disks it will show you eight drives, if it's on top of the RAID only one. Edit - And if there's no zpools at all, then you're sharing the device directly.

Regarding saving the data ... assuming the underlying RAID6 isn't hosed you should be able to replicate the iSCSI configuration but if it's a single drive to ZFS it might consider it dirty.

Are they configured as mirrored dynamic disks to the connected Win2008 server or a stripe?

 

[jgreco]

@jgreco mfiutil is the FreeBSD utility for managing MegaRAIDs, if there's anything defined there it's using the hardware RAID.

I suspect I know that, since I talk about it in the LSI sticky, and also I already told him it was hardware RAID. Good morning and welcome to the discussion ;-)

run "zpool status" from a shell. If it's directly on the disks it will show you eight drives, if it's on top of the RAID only one.

It's not going to show him eight drives. It clearly shows a single volume configured.

Regarding saving the data ... assuming the underlying RAID6 isn't hosed you should be able to replicate the iSCSI configuration but if it's a single drive to ZFS it might consider it dirty.

I have a suspicion there's no ZFS in the mix. We had a few people who just wanted to use FreeNAS to export raw storage via iSCSI, leaving ZFS entirely out of it, and it was possible to create an iSCSI configuration that used a raw disk device (which is what mfid0 appears to be to FreeBSD, even if it is actually a RAID volume of multiple drives in reality).

This should in theory work just fine but I don't know if it is supported in FreeNAS 9.

 

[HoneyBadger]

Knee-jerk reaction was that this was yet another ZFS-on-RAID solution since the OP inherited it from a prior admin.

I also don't often see straight iSCSI on devices done, at least not on a system that offers ZFS, if it was a Linux+SCST setup sure. After going back and rereading I now suspect that's the case.

I'd say that if the power went down mid-write the RAID volume got corrupted somehow. Does the PERC boot ROM give any logs or insight into this?

 

[Randolfini]

Thanks guys for the quick replies.

@HoneyBadger
[root@100NAS01 ~]# zpool status
no pools available

@jgreco
I think you may be right that there is no ZFS. There is one volume from the RAID controller, RAID6, and that was shared out via iSCSI as a device extent RAID6 Disk /dev/mfid0 to the Windows File Server.

@HoneyBadger
The PERC RAID shows optimal / healthy, no indication that the raid is failed or degraded.

Try 1: I re-installed FreeNAS-8.3.0-RELEASE-x64 (r12701M) on a new USB drive and re-configured settings manually. When I presented the mfid0 device extent via iSCSI it did not auto-mount in Windows. I assume because the iSCSI serial number was different? In any case the LUN was un-initialized and any further action would have wiped it...so I stopped.

Try 2: I re-installed FreeNAS-8.3.0-RELEASE-x64 (r12701M) to the USB drive, imported the configuration from an identical NAS, 100nas02, and changed the IPs, IQN, etc to match 100nas01. I presented it back to the Windows host with the same result. It saw the LUN as un-initialized

Presently I am in configuration Try 2.

Not sure if the below helps.

[root@100NAS01 ~]#dmesg | grep mfi0 | more
mfi0: 10705 (boot + 3s/0x0020/info) - Firmware version 1.22.32-1371
mfi0: 10706 (boot + 23s/0x0008/info) - Battery Present
mfi0: 10707 (boot + 23s/0x0020/info) - Controller hardware revision ID (0x0)
mfi0: 10708 (boot + 23s/0x0020/info) - Package version 6.3.1-0003
mfi0: 10709 (boot + 23s/0x0020/info) - Board Revision
mfi0: 10710 (boot + 37s/0x0004/info) - Enclosure PD 20(c None/p0) communication
restored
mfi0: 10711 (boot + 37s/0x0002/info) - Inserted: Encl PD 20
mfid0: <MFI Logical Disk> on mfi0
mfi0: MFI_DCMD_PD_LIST_QUERY failed 2
mfi0: 10712 (boot + 37s/0x0002/info) - Inserted: PD 20(c None/p0) Info: enclPd=2
0, scsiType=d, portMap=09, sasAddr=5782b0b0254d6300,0000000000000000
mfi0: 10713 (boot + 37s/0x0002/info) - Inserted: PD 00(e0x20/s0)
mfi0: MFI_DCMD_PD_LIST_QUERY failed 2
mfi0: 10714 (boot + 37s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: enclPd=20
, scsiType=0, portMap=00, sasAddr=1221000000000000,0000000000000000
mfi0: 10715 (boot + 37s/0x0002/info) - Inserted: PD 01(e0x20/s1)
mfi0: MFI_DCMD_PD_LIST_QUERY failed 2
mfi0: 10716 (boot + 37s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info: enclPd=20
, scsiType=0, portMap=01, sasAddr=1221000001000000,0000000000000000
mfi0: 10717 (boot + 37s/0x0002/info) - Inserted: PD 02(e0x20/s2)
mfi0: MFI_DCMD_PD_LIST_QUERY failed 2
mfi0: 10718 (boot + 37s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info: enclPd=20
, scsiType=0, portMap=02, sasAddr=1221000002000000,0000000000000000
mfi0: 10719 (boot + 37s/0x0002/info) - Inserted: PD 03(e0x20/s3)
mfi0: MFI_DCMD_PD_LIST_QUERY failed 2
mfi0: 10720 (boot + 37s/0x0002/info) - Inserted: PD 03(e0x20/s3) Info: enclPd=20
, scsiType=0, portMap=03, sasAddr=1221000003000000,0000000000000000
mfi0: 10721 (boot + 37s/0x0002/info) - Inserted: PD 04(e0x20/s4)
mfi0: MFI_DCMD_PD_LIST_QUERY failed 2
mfi0: 10722 (boot + 37s/0x0002/info) - Inserted: PD 04(e0x20/s4) Info: enclPd=20
, scsiType=0, portMap=04, sasAddr=1221000004000000,0000000000000000
mfi0: 10723 (boot + 37s/0x0002/info) - Inserted: PD 05(e0x20/s5)
mfi0: MFI_DCMD_PD_LIST_QUERY failed 2
mfi0: 10724 (boot + 37s/0x0002/info) - Inserted: PD 05(e0x20/s5) Info: enclPd=20
, scsiType=0, portMap=05, sasAddr=1221000005000000,0000000000000000
mfi0: 10725 (boot + 37s/0x0002/info) - Inserted: PD 06(e0x20/s6)
mfi0: MFI_DCMD_PD_LIST_QUERY failed 2
mfi0: 10726 (boot + 37s/0x0002/info) - Inserted: PD 06(e0x20/s6) Info: enclPd=20
, scsiType=0, portMap=06, sasAddr=1221000006000000,0000000000000000

 

[jgreco]

Knee-jerk reaction was that this was yet another ZFS-on-RAID solution since the OP inherited it from a prior admin.

Geez, pay attention. You didn't see me do the exact same thing above?!?!?

I'd say that if the power went down mid-write the RAID volume got corrupted somehow. Does the PERC boot ROM give any logs or insight into this?

Nah, very unlikely in the grand scheme of things. You don't get uninitialized volume from that.

OP, I'm going to suggest that you might want to see if you can recover the backup config off the old USB.

See for example https://forums.freenas.org/index.php?threads/recover-freenas-configuration-from-os-drive.15426/ but look instead to see if there's a usable .bak file.

iSCSI has some weirdness with serial numbers and other stuff where things get weird when you try to move the disk to a different server. I don't recall the specific things that it has to see, but I can tell you that your best bet is getting access to the old USB's exact configuration and duplicating that. The .bak file on the old USB is a good shot at that, as would be the istgt.conf file over in /etc/local/istgt/istgt.conf

 

[Randolfini]

OP, I'm going to suggest that you might want to see if you can recover the backup config off the old USB.

See for example https://forums.freenas.org/index.php?threads/recover-freenas-configuration-from-os-drive.15426/ but look instead to see if there's a usable .bak file.

iSCSI has some weirdness with serial numbers and other stuff where things get weird when you try to move the disk to a different server. I don't recall the specific things that it has to see, but I can tell you that your best bet is getting access to the old USB's exact configuration and duplicating that. The .bak file on the old USB is a good shot at that, as would be the istgt.conf file over in /etc/local/istgt/istgt.conf

Unfortunately I tried restoring the freenas_v1.db already. There are bad blocks on the USB stick that corrupted it and the .bak. I tried both actually. I even opened both in a SQLLite program and the structure is gone when compared to a known good freenas_v1.db.

I just tried looking for the /etc/local/istgt/istgt.conf and only see the samples. I mounted /dev/da1s1a to /var/tmp/recovery.

[root@100NAS01 /var/tmp/recovery/etc/local/istgt]# ls -l
total 33
-rw------- 2 root wheel 922 Oct 24 2012 auth.conf.sample
-rw-r--r-- 2 root wheel 5124 Oct 24 2012 istgt.conf.sample
-rw-r--r-- 2 root wheel 8942 Oct 24 2012 istgt.large.conf.sample
-rw------- 2 root wheel 733 Oct 24 2012 istgtcontrol.conf.sample

 

[jgreco]

I figured that might be the case since istgt.conf gets built upon reboot, and might not be persistent anyways, but never hurts to check. You might also try looking at the binary contents of the db file to see if you can identify anything useful, probably look in the neighborhood of the IQN.

 

[Randolfini]

I tried once and didn't find it, however I was searching for just the field name in the GUI...Serial. I reviewed the config of a working db file and found it is next to the iSCSI target name. Once I realized this, I searched the corrupt db again and found it:

Code:

Auto raid6782bcb2a6fd000100NAS01DiskAutorw

I unmounted and disconnected the iSCSI connection in Windows then fixed the iSCSI device serial. I represented and still comes up uninitialized : (

I am using GetDataBack for NTFS and it sees the partition after a scan, I can see the files. I'm just going to use this to recover 3/6-3/11, the gap from my last full backup. It appears the GPT partition is the problem.

I've tried a slew of apps but nothing recovers the Dynamic GPT correctly. I saved the existing partition scheme and have had to revert a couple of times after trying different things.

I think this is the end of the road for this one. Thank you so much jgreco and HoneyBadger for your quick responses and expertise.

 

[jgreco]

We're sorry it didn't work out. This stuff can be a little more convoluted and complexicated than you'd want, when you're trying to figure out what broke. :-(