Failed resilver required mounting zpool as read-only; can't mount iSCSI extent

[r3mus]

Long story short, I've got a 12TB zpool (raid-z) that died when resilvering from a disk failure. Cause isn't important, only the fact that the data appears to still be intact.

Using zpool import -R /mnt/tank -o readonly=on rz0 I was able to successfully mount the "rz0" zpool in /mnt/tank. Once that worked, I can now browse into that directory and see "ex0", my iSCSI file extent, sitting there. If I run file ex0 I get this:

[root@freenas] /mnt/tank/rz0# file ex0
ex0: x86 boot sector; partition 1: ID=0xee, starthead 0, startsector 1, 4294967295 sectors, code offset 0xc0, OEM-ID " \320\274", Bytes/sector 190, sectors/cluster 124, reserved sectors 191, FATs 6, root entries 185, sectors 64514 (volumes <=32 MB) , Media descriptor 0xf3, sectors/FAT 20644, heads 6, hidden sectors 309755, sectors 2147991229 (volumes > 32 MB) , physical drive 0x7e, dos < 4.0 BootSector (0x0)

So it appears that the data is at least partially intact.

Problem:

I can't mount that file extent through the web GUI, and there doesn't seem to be any way to force it to mount. When trying to do it through the GUI, I'm getting this error: Your path to the extent must reside inside a volume/dataset mount point; all the information on the web points to how to create a new extent, but not mount an existing one that doesn't show up.

Solutions:

I need to A) find a way to mount this extent within FreeNAS, or B) (preferably) find a way to re-establish the iSCSI extent and target so I can transfer directly off the underlying partition (it's Windows NTFS).

 

[r3mus]

For some context, here are some screenshots:

This one shows the extent flat file in the /rz0 mounted directory (read-only)


This one shows the browse feature not showing any of the filesystem at all


And this one shows the error message when trying to type the path manually.
According to the documentation typing the path to an existing extent and setting the size to 0 should work, but alas, it doesn't.


There is another thought I had -- is there a way to force an existing array to load in read-only mode on bootup? This whole thing started because I couldn't get the original OS disk to load without a fatal kernel panic. if I could set the array to load as read-only instead, I could use the original iSCSI configuration to make it work?

 

[r3mus]

Handy links for importing and mounting a zpool as read-only (for reference):

http://forums.freebsd.org/showthread.php?t=17560 and http://forums.freebsd.org/showthread.php?t=34986

When combined, they explain how you can use zpool import -o readonly=on -R /<mountpoint> <poolname> to mount a damaged array as read only.

 

[r3mus]

OK! I've solved this bizarre one. I'll document the solution here as best I can in case anyone else ever has the same issue.

Again, the back story is that I replaced a failing disk with another new disk. The resilver failed almost immediately (after only transferring 6MB), after which a kernel panic caused the OS to crash instantly. It also continued to crash each time on bootup unless ALL member disks of the zpool were physically disconnected.

In hindsight, had I been able to edit the zpool configuration to not auto-import the raid-z, perhaps I would have been able to salvage the original OS. But I wasn't able to do that, and thus ended up doing this:

1. Booted up to the original FreeNAS install with all zpool member disks physically disconnected.
2. SSH'd into the box and made a copy of /etc/local/istgt/istgt.conf. I used notepad++'s NppFTP plugin to do this really easily.
3. Powered down the NAS again and replaced the original OS disk with the fresh FreeNAS install disk (yes, replace the disk, don't install over top or have both connected simultaneously).
4. Booted up to the clean FreeNAS install with all zpool member disks except the NEW replacement disk reconnected.
5. Ran zpool import -o readonly=on -R /mnt rz0 (where rz0 is the name of my particular zpool). This mounted the zpool as a read-only pool, preventing the kernel panic.
6. SSH'd into the box again and copied my backup of /etc/local/istgt/istgt.conf onto the fresh install, replacing the existing one. I had to make one major change: I changed ReadOnly No to ReadOnly Yes under [LogicalUnit1]. This was necessary since the zpool is read only, and the istgt service wouldn't start until I did this.
7. I restarted istgt with service istgt restart.
8. My iSCSI target was then made available and I saw my initiator client (a Windows box) reconnect and mount the partition.
9. Then things went a little wrong. I discovered that the partition that mounted was RAW not NTFS, implying that the partition configuration data was lost. I played around with diskpart and some other commands, but to no avail. Even chkdsk /f failed with a "Device I/O" error.
10. On a whim, I decided to plug in the OLD fail(ing) disk to the array. FreeNAS immediately picked it up, and running zpool status showed me that the old disk was automatically added back into the pool.
11. Then not 5 minutes later, the partition refreshed on the Windows box and everything came back.

In even further hindsight, it may have been as simple as plugging in the old fail(ing) disk into the array on the original FreeNAS install. I'd love to fiddle around and test this theory, but since it's working and the data is available through iSCSI, I'm not going to touch anything until I've transferred all the data off the array.

Hopefully someone else who has an issue like this will find this post helpful - good luck!