[RESOLVED] Identifying which disk is in a VMDK

[liteswap]

I hope someone is going to say that the answer to this problem is obvious...

Scenario: FreeNAS running under ESXi 5.0 (so I can run other VMs on the same box, obviously).

Storage: One SSD, containing the FreeNAS OS as a VM. Six 3TB SATA disks, each containing two vmdk virtual disks, inside each of which is a ZFS virtual disk and file system, managed by the FreeNAS VM, and containing two zpools of ~12TB and ~3TB raidz striped across all six disks.

It's not an ideal setup, I know, so I plan to migrate the smaller of the two zpools to a pair of separate mirrored disks, and migrate the 12TB zpool onto the remaining disks using direct access, one at a time, and allowing the zpool to resilver. Yes, I have a backup.

My question is this: how can I discover which of the FreeNAS disks is contained in which vmdk? Using the Web GUI and the terminal, I get two different identifications for each vdev. The Web GUI calls them da1, da2....dax. The 'zpool status' command shows this (the other pool is the same for this purpose):

Code:

NAME                                            STATE     READ WRITE CKSUM
        data                                            ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/9bf6eb4d-df8c-11e0-8143-000c29e91528  ONLINE       0     0     0
            da7                                         ONLINE       0     0     0
            gptid/9c8f6085-df8c-11e0-8143-000c29e91528  ONLINE       0     0     0
            gptid/9cda300f-df8c-11e0-8143-000c29e91528  ONLINE       0     0     0
            gptid/9d27ef95-df8c-11e0-8143-000c29e91528  ONLINE       0     0     0
          gptid/42f8319d-f70b-11e3-8c9e-000c29e91528    ONLINE       0     0     0

Only da7 (for a reason I don't quite understand) corresponds to the Web GUI's name for one of the vdevs. And I see no correspondence between any of the names and the vmdk files (unsurprisingly) but can't find out which FreeNAS disk is in which vmdk. 'zdb list' doesn't help.

Is it a case of pulling a disk and seeing which vdev ZFS reports as missing?

Thank you.

 

[SweetAndLow]

You are probably not going to get any responses because you are visualizing Freenas. Your configuration is absolutely ridiculous and I had to read your post 3 times to even begin to understand what you did. I also don't think you are going to be able to migrate your data off because to do direct access you are going to have to let FreeNas use all of the disk so you can't share it with the vmdk during the transfer, maybe you have 6 more drives you can use instead.

As for finding out which disk a vmdk lives on from inside freenas, that isn't going to be easy. You might be able to look at the vm's settings and compare the disk in there and their location to what shows up in freenas. They are orginized with SCSI(0:[0-9]) which might possibly map directly to the numbers freenas uses, this would be luck though.

 

[liteswap]

Yes, I understand I will have to give all the disk to do direct access. After migrating the smaller of the two pools off, I reckon I have 2 options:
1. Vape the main pool, set up raw disk access and rely my backup - which leaves me for some time with only one copy of the data.
2. Set up raw disk access, one disk at a time, and let ZFS resilver the raid - one disk at a time.

And I know the setup doesn't make much sense but it seemed to at the time. Now I realise it doesn't. hence my desire to be rid of it.

 

[mjws00]

You see that you have one single disk striped with 5 in z1, right? If that single disk fails... or you pull it to set up raw disk access you lose the pool. No way to pull a disk to resilver there is no redundancy on the last disk.

Finding which disk is mapped where isn't hard. On a FreeNAS CLI
#camcontrol devlist **will show you where things are mapped and the device.
#glabel list **will show you what is up with your partitions. And their name which is mapped to your pool.
#glabel status ** is more terse.

From the vsphere client. You can see the properties of the disk SCSI(0:0) etc... and its filename on the datastore.

'Not ideal' is a pretty significant understatement. However, if you manage to pull off migration to RDM. You could migrate further to BAREMETAL. Seems to me you are sitting on a time bomb, and I probably shouldn't be answering these questions. You have pool problems in addition to your vm issues, and your target modality will probably sink your data. RDM is a very poor choice.

I don't mind watching it all unfold. You said you have a backup.... I'd make sure it's verified.

 

[liteswap]

You are of course dead right. My setup is even more screwed than I thought, having committed the error of attaching a new drive and assuming (poor fool) that it was part of the raidz not just striped.

I have now read cyberjock's guide- something I should have done much much earlier. Time to consider a way out of this mess... A message to others: learn from my mistakes.

Thank you for the CLI commands.

 

[mjws00]

The upside is you have backups. Many do not.

The only out is to move the data off the pool and rebuild it. You haven't listed hardware or anything... so I don't know your options or what appropriate risk is. I prefer to stick with the "do not virtualize" mantra cited often. It seems to cause far less heartache :) However, I'm always curious about failure modes and appreciate good analysis of a disaster. Hopefully without REAL loss.

Good luck. Glad you caught this before you were burned.

EDIT: One option you have that others don't. You can shuffle your vmdk's. They aren't damaged yet. Might be easy to grab a large disk or two and add a datastore in order to do some shuffling and free things up. Moving many TB always sucks even locally.

 

[cyberjock]

* has a heart attack after reading the first post *

 

[mjws00]

Heh. I could almost see your face as I was reading. But couldn't resist the trainwreck ;)

 

[liteswap]

Awww, gimme a break guys :)

A plan has percolated down:
1. Get the data off the small 3TB pool onto a new 3TB disk. A single disk is not a problem here as it's backup not primary data.
2. Double-check I can read from the backup (currently on a 12TB Seagate NAS)
3. Vape the six disks, & create a raidz2 pool with raw access (is this best?)
4. Restore from backup (how long is that going to take? :))
5. Copy from the 3TB disk to the big pool (no need for a separate pool for backups really)
6. Vape the single disk and add it to the raidz2 set as a live disk or a hot spare.

Train wreck averted?

 

[SweetAndLow]

You can't do this

6. Vape the single disk and add it to the raidz2 set as a live disk or a hot spare.

adding drives to a vdev doesn't work. You can only create more vdevs. This is how you got the single disk strip in the first place in your original pool. If I was you I would just not use Virtualization with freenas. If you are determined to virtualize then use a different operating system. You have lots of reading and learning to do.

 

[mjws00]

So here is the deal. I am fully on board with the school of hard knocks. If you pull off the transition, you will have demonstrated at least the fact that you have a legit backup. The warnings you see EVERYWHERE, and the fightclub model of virtualization are real. Read everything by jgreco, DO NOT VIRTUALIZE THREAD. Understand I Wish to Suffer and Do it Anyway what would a pro do? It is always about recovery modes.

Many, many, many people do stupid things with virtualization and get stuck in a situation exactly like yours. Only they dropped a disk and are already screwed, or they rebooted and discovered their pool was unavailable. Even though they did everything right and it worked great for a year. Instantly everything is GONE. There is no recovery, and the best experts can't fix it. The correct and only answer is rebuild the pool restore from backup.

These losses make some people sad, especially the more sensitive mods like cyberjock (tongue firmly in cheek). I am a jaded sob and don't get paid to protect you from yourself. I'll write a wall of text. Warn you. Let you know the gun is loaded. EVEN USE CAPS. But after that... It's on you. In addition I would love to see virtualization stable, and a known risk factor. So that requires breaking a few eggs. Your eggs. ;)

Even after all that I won't say it's ok to run virtualized. I will say I trust jgreco's opinion. When you say 'raw access' in esxi we would normally think of that as RDM or raw device mapping. RDM on local storage is unsupported by vmware. It's a hack. In addition we've seen corruption happen with that model. Might just be a partition table you can use wizardry to recover from... but if it can corrupt that, then it can clobber zfs metadata and screw you.

If by 'raw access' you mean. Passing a known good controller model through, on great server hardware, with lots of resources, via vt-d. Perhaps you've read something. You may have some outs from that position, and you can instantly be running and recovering on baremetal. You may also just lose everything without warning. The problem is you'll be on your own. The wizards, may or may not play. The autopsy is ugly. There is no crying.

tl;dr This is SPARTA!!!

 

[liteswap]

mmm, yes, take your point. Have read jgreco's virtualisation post - most helpful, thank you.

But I am running ESXi 5.5 on an Intel S1200BTL server-grade board with a Xeon 31230 + 16GB ECC RAM, all allocated to FreeNAS. Controller pass-through is then possible - and I seem to have passed through most of gates set up in the 'don't virtualise' thread....

 

[mjws00]

Those are good gates to have passed.

 

[liteswap]

Thank you!

So now the plan is:
1. Check backup exists.
2. Check backup is restorable
3. GOTO 1. Until 100% sure
4. Export both pools
5. Set ESXi to pass-through the Intel board controller (or buy an IBM ServeRAID M1015 card and use that instead?).
6. Attach a 1.5TB disk I have lying around in order to accommodate the other VMs
7. Vape both ZFS pools
8. Build a 7 x 3TB disk raidz2 pool
9. Cross fingers
10. Restore data
11. Go to the pub

 

[mjws00]

If you can grab the m1015 I would do it that way. Only because it is known to work well. Many things that work in theory don't always pan out via vt-d. Flash to it mode p16. Yes exact firmware version matters.

The Intel controller may be fine... but it could have a driver issue or glitch that is unknown. You aren't really set to test it for a month. So I would play it safe... The intent of course is to minimize risk where we can and align ourselves with the largest pool of user experience and testing.

 

[liteswap]

An M1015 is on its way (thanks eBay), along with a couple of SAS break-out cables. We shall see how this works. Reportback when the new hardware arrives, assuming my disks don't flake in the meantime...