Ok it sounds like Fln & VT have a fairly good idea of what they are doing. The really important thing to know is make sure you are using physical RDMs and not virtual ones. Virtual is indeed the kiss of death. Physical ones are fine and there is no reason to fear for your data if your drives are passing in unique serial #s which I saw someone verify that they where for at least one of the installs. The only real danger is vmware might remove or break the function in 6.0 or beyond since it's not exactly a supported method in their eyes, RDM was designed for passing though LUNs for fiber channel SANs, see below for more comments.
So go find the threads where people had a RAIDZ2, one disk failed, so they replaced it, and suddenly 3 other disks are "CORRUPT" and their zpool is unmountable. Cause that's exactly what has happened to quite a few people that have tried to do RDM and then showed up in the forum to tell us mods and VM gods that we're morons because it works great. This is the principle reason why RDMs are so strongly advised against. It sounds great, and it works great, until it doesn't.
Also it looks like the root cause has been tracked down to a known FreeBSD bug. I'd suggest whoever wants to champion this issue go ahead and register at http://support.freenas.org and create a ticket with the important details posted in this thread. Several of the ix guys are also FreeBSD developers and can possibly poke the someone who is responsible for the broken code. I'd then monitor FreeBSD and watch for the bug fix, it would be nice if they'd fix it before 9.2 ships(which is very soon). If you can find when they patch FreeBSD and what the MFC # is for the patch you can probably then get the FreeNAS guys to pull it in for the next FreeNAS release. In the mean time stick to the 8.3.x series until this gets straightened out. Also please use LSI SCSI or SAS(I'd probably try SAS though SCSI might be a safer choice) has your controller type, buslogic might work but that's also looking for trouble down the road.
*cough*, those people that I mentioned above that had RAIDZ2s that failed.... they were using 8.x...So don't tell me that after this 9.x bug is fixed all will be better. Remember, some of us mods read virtually every thread created, so we see what people do wrong regularly and what people do right. And let me tell you from person experience, those people that lose data really stick in your mind. We always want to know whether it was stupidity(like a 10 drive striped zpool), neglect(turned it on and never looked at it again), or an actual issue that needs to be fixed.
It's been well hashed out now that RDM isn't the best way to fly but some days it is the only way to fly.
Yeah, if you plan to fly with your gas tank already on "E". Then sure, go ahead. But what goes up will come down...in a ball of fire.
IThat said I've successfully run a production server using RDM physical disks(it just holds my backups so I'm not loosing sleep at night - but the backs do put some real stress on the VM & FreeNas during the backup session), it wasn't my 1st choice but I had no other choice. The server had a adaptec RAID controller that I could pass through but it had issues and it was not in my control to replace it with a desired LSI card. Using an adaptec controller with FreeBSD is probably worse then RDM pass through if you love your data(if you don't believe me read the source code for the adaptec controller). Eventually I was able to build myself an adaptec driver from source the worked ok and I switched from RDM to using that, just because I feared someday vmware would nuke RDM, that and I like being able to bring up my ESXi boxes from a fresh install without spending and bunch of time in the cli building RDM files to bring my SAN online.
Well, good for you. And unfortunately, you're that person I mentioned above that shows up, says its worked great for so long, but you'll eventually lose everything. And then you'll be back in this thread advocating against it like most RDM-ers have done. You'll go back and edit your posts to make sure everyone understands that it just doesn't work and your advice and recommendation has made a 180.
Notice that STILL none of the VM gods in the forum have posted...But good luck. Gonna unsubscribe to this thread thread now. I get pissed just listening to the ignorance of some people and how they really think they know better than the guys that have been here since 8.0.