Disks Not Configured in FreeNAS 9.1 Release

[VTecheira]

Couple of caveats and disclaimers...

First, forgive me if this is a double post but I didn't find anything to match my situation... The closest I came was this post but it didn't match exactly.

Second I am running this as virtual machine on ESXi 5.1 (with the July 2013 patches). With 6GB of RAM allocated and 1 vCPU with 3 cores.

I recently set up my system to be RAIDZ1 with 5 2TB HDDs. It seemed to go fine but I was having some CIFS and NFS weirdness. Since 9.1 was about to be released I jumped in on RC2 to find that I couldn't import my zpool. The root cause of this seemed to be that my disks are properly being detected in 9.1. Going back to 8.3.1 they get detected fine.

The geometry seems to be failing for the disks in 9.1 (and problem persists even in the release). Since I hadn't committed much data yet, I cleared everything out and started with 1 2TB disk. But even that isn't getting detected.

Here's what geom disk list shows under 9.1.

Code:

1. Name: da0
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Mode: r2w1e4
  descr: VMware Virtual disk
  ident: (null)
  fwsectors: 63
  fwheads: 255
 
Geom name: da1
Providers:
1. Name: da1
  Mediasize: 0 (0B)
  Sectorsize: 0
  Mode: r0w0e0
  descr: ATA WDC WD20EARS-00M
  lunid: ATA    WDC WD20EARS-00MVWB0                        WD-WMAZA1194853
  ident: WD-WMAZA1194853
  fwsectors: 0
  fwheads: 0

Here's the dmesg for those drives under 8.3.1

Code:

da0 at mpt0 bus 0 scbus2 target 0 lun 0                                   
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device           
da0: 300.000MB/s transfers                                                 
da0: Command Queueing enabled                                             
da0: 2048MB (4194304 512 byte sectors: 255H 63S/T 261C) 
da0: Command Queueing enabled                                             
da0: 2048MB (4194304 512 byte sectors: 255H 63S/T 261C)                   
da1 at mpt1 bus 0 scbus3 target 0 lun 0                                   
da1: <ATA WDC WD20EARS-00M 51.0> Fixed Direct Access SCSI-5 device         
da1: 300.000MB/s transfers                                                 
da1: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)  

I tried setting the ashift value, but it didn't really seem to have any effect I can detect.

Please advise.

[Edit]
I forgot to mention a few other things. The disks are exposed to the VM as RDM physical drives. I switched from LSI Logic SAS to LSI Logic Parallel on the off chance that seems to be issue.

 

[cyberjock]

RDM with FreeNAS is a recipe for disaster. You should give up right now with trying to use RDM if you value your data at all. When RDM fails you, you'll get no sympathy from the forum. If you value your data that little, just delete it now and accept the loss.

In case you missed it, this is thoroughly discussed in the threads Absolutely must virtualize. A guide to not completely losing your data and Please do not run FreeNAS in production as a virtual machine. Those threads were created because so many people have lost everything for various reasons(often outside of your control). What you should understand though, is that you shouldn't expect much of anyone to help you because its pretty heavily frowned upon trying to run FreeNAS in a VM except to experiment. Even more so if you don't follow the warnings and recommendations that make using it in a VM "safest"(if there is such a thing). Pretty much if you aren't following the minimal recommendations of that thread you can expect almost no answer because most everyone that has a clue doesn't want to give the impression that they are endorsing it later.

Besides, for many people, they'll blame FreeNAS when(not if) they lose their data despite the warnings. It's happened before, even people that said that they understood it wasn't supported would still show up and flame the forum with their sob story(dog ate my homework, wife wants a divorce, can't afford the next truck payment, etc.) and most of us don't want to hear it again.

Not only that, RAIDZ1 "died" in 2009.. so why you think that gives you data protection in 2013 is beyond me. There's a link in my sig if you want to see for yourself.

 

[FlynnVT]

I saw the same behaviour on ESXI 5.1 with up-to-date patches etc. FreeNAS 9 detects all physical RDMs as 0 bytes, despite having read the actual disc Make & Model OK. The same setup worked well on all iterations of FreeNAS 8.

I tried many things. In particular:
1) This had no effect (post #16): http://hardforum.com/showthread.php?t=1731949
2) Changing the presentation of RDMs though to the VM had no effect: LSI parallel/serial both for the adapter type in the VM's and by editing the RDM definition directly.
3) Creating a virtual (-r) rather than physical (-z) RDM did work OK. The disc content was detected and accessible. Unfortunately this route loses the ability to read SMART info, spin down discs etc. I also think it breaks RDMs > 2Tb (not sure if this limit has been removed on more recent versions of ESXi/VMFS) and generally moves further away from having physical discs that I can transplant to a real (non-virtualised) FreeNAS instance at a moments notice.

Is this caused by a change in the FreeBSD 9 CAM?: http://sourceforge.net/p/nas4free/bugs/65/

Before anyone (else) says ESXi + FreeNAS = bad. Yes, I know and I've read the warnings. This is only a home server and I like the convenience, features and basic veneer of redundancy that multi-disc ZFS Z1 gives. (What do you mean "RAIDZ1 "died" in 2009"?? Single disc redundancy is still single disc redundancy. I think it's a reasonable setup for a three disc tank). All my data is periodically backed up to non-ZFS external discs. In short: ESXi suits my other usage and FreeNAS is a great system.

 

[VTecheira]

CyberJock -- I've read the warnings on FreeNAS and RDM and until now I hadn't really committed any data to FreeNAS exclusively. So at this point it really is experimenting and I know I will have a long cry if I commit data and something goes wrong. I've tried to follow the recommendations with the exception of PCI passthrough (since I don't have that).

I've also read the post that RAIDZ1 is dead, but I also know that RAIDZ1 was recommended for 5 disks. My plan was to actually go with whatever version 9 mentioned (even if it meant losing a lot of data to parity). Once again I recognize I assume all risk. Basically I don't have a budget for dedicated hardware, and it's either ESXi or bust for me.

FlynnVT --
It looks like you were in the same boat, and that bug seems to be the best lead. As an aside I tried the NAS4Free last night, and I have a richer appreciation for FreeNAS' rebuilt UI.

All that aside unless we are saying the FreeNAS 9.1 doesn't support RDM disks for experimenting then I believe my original question is still valid (even if it assumes extra risk).

 

[VTecheira]

Dang it... I lost my forum post (I jumped to another link before I realized I hadn't posted my reply)... So here's the short version.

FlynnVT -- Yep, it definitely seems you were in the same boat as I as was; and that bug link you posted seems to be the best lead.

CyberJock -- I've read and understood the caveats on both FreeNAS in ESXi and RAIDZ1. I was going to have 9.1 make the recommendation for me since I know that for 5 disks the recommendation is RAIDZ1. Unfortunately I don't have the resources to go completely physical, and I have to live with that risk. It is ESXi or bust. Since I haven't committed any data yet we can consider this experimentation. (BTW if it wasn't obvious this is for a home setup not a business env)

Unless we are saying that FreeNAS 9.1 does not allow RDM experimentation, I believe the original question still stands (all risk assumed).

[Edit]
After reading the virtualization post again (for the 20th time maybe?) I got the idea to see if I can go completely physical via USB (temporarily) to confirm this is an RDM only issue. I know it won't be an exact match but it'll provide some insight.

[Edit II]
Another thought just occurred. Could this have something to do with ZFS v5000?

 

[FlynnVT]

My guess was something in the long chain of SCSI/ATA emulation between ESXi and the FreeBSD CAM.

It isn't ZFS specific and should effect UFS too, as you can see from your 0 byte geom. It shouldn't effect USB-passthrough discs as that emulation is across a generic EHCI layer (itself emulated, but at a level below the disc-level command flow). The FreeBSD CAM will be speaking directly to the embedded processor in the USB-SATA bridge.

Maybe this is a deficiency in ESXi: the RDM mechanism or emulated LSI SCSI controller could itself have a bug. The sensitivity to having virtual or physical RDMs proves that the abstraction isn't perfect. Which ever side is at fault, FreeBSD 8 seems to have done things in a different way that didn't excite such a shortcoming.

Now, which side is more likely to make an adjustment sooner? ;-)

I found a forum post before where someone claimed that ESXi 5.0 (or 4.x) had less issues with RDM passthrough, but can't find it again.

VT-d isn't an option for me either so I'll be sticking with FreeNAS 8.3 with physical RDMs under ESXi 5.1 for now. Both are otherwise rock solid.

 

[VTecheira]

[Another update]

Not that there was much doubt, but going completely physical FreeNAS 9.1 picked up the disks with no issues whatsoever. So the problem clearly has to do with the ESXi/RDM/FreeNAS interaction.

 

[hervon]

Guys i'm in the same boat as you are with no solution so far. Going completely physical also worked fine with FreeNAS 9.1. Me too, i'll stick to 8.3 (that works great for me) until there is a solution. I may eventually upgrade to a CPU that supports VT-d.

 

[cyberjock]

Creating Raw Device Mapping (RDM) is not supported for local storage (1017530)

 

[cyberjock]

That will explain random unexplained behavior WRT RDM. :)

 

[VTecheira]

Hi Cyberjock... I read the article, the lack of support specifically has to do with the absence of a globally serial number. Since the RDMs were manually created I believe that issue has been circumvented. Furthermore since the functionality can be manually enabled, by Allowing VMware ESX 3.5 to use Raw Device Maps (RDM) on Egenera pservers (1014513) I'm not convinced that's the root problem, especially since we don't see this behavior in 8.3.1. Now it's possible that 9.1 has code executed that specfically requires that ID but it seems unlikely. If that were the case I don't think the disks would even be detected, rather than detecting the name/id of the disk and just failing on the geometry and size.

 

[cyberjock]

Hint: it is NOT circumvented by manually creating them. You definitely do not understand that post. The issue isn't with the guest OS, it's with the host and how the hardware interacts with ESX. Do not be fooled. Plenty of other people were fooled, and they lost everything when their RAIDZ2 had a single drive fail despite doing everything else as right as possible.

That may not be your problem specifically.. I stopped paying attention to people's individual problems the second they mentioned they are doing things with virtualization that I consider "less than smart". And not as a coincidence, nobody else has either. But RDM is not as reliable as you think.. and you'll probably figure it out after its too late.

Good luck!

 

[FlynnVT]

That may not be your problem specifically. I stopped paying attention to people's individual problems the second they mentioned they are doing things with virtualization that I consider "less than smart".

That would explain your responses!

Look, I understand the anti-VM stance and the reasons behind it. Bare metal FreeNAS beats a VM in many ways, particularly when using RDM instead of VT-d.

However, simply reiterating that point of view does nothing to solve the problem being discussed on this thread.

-- EDIT --
Some other related posts:
ESXi 5.0 having better RDMs: http://support.freenas.org/ticket/1977

FreeBSD breaking RDMs between 9.0 and 9.1: http://lists.freebsd.org/pipermail/freebsd-bugs/2013-January/051368.html
...the open PR for the FreeBSD project: http://www.freebsd.org/cgi/query-pr.cgi?pr=174908

Ditto for NAS4Free: http://forums.nas4free.org/viewtopic.php?f=16&t=1020 (Also says that ESXi 5.0 fails in the same way, that -r instead of -z works, that it depends on the physical SATA controller type, and that changing the VM SCSI controller to 'buslogic parallel' worked [I haven't tried this, but do remember vSphere strongly suggesting LSI rather than BusLogic for FreeBSD guests])

Again, it seems something changed in FreeBSD 9.1 that exposed a bug in ESXi. The "-r" and BusLogic options may be workarounds for now, but future updates to ESXi and/or FreeNAS could change the situation again.

 

[VTecheira]

Flynn, you've once again summed up my sentiments. Cyberjock is clearly quite knowledgeable and obviously anti-FreeNAS in VM. That said I'll maintain my FreeNAS 9.1 VM to help investigate if anyone has any other quest. Meanwhile it's back to 8.3.1 for me (and maybe some napp-it experimentation as well).

Cyberjock, you are the elder statesman here, so I'll ask: Is it worthwhile to open an official FreeNAS bug, or is this a scenario that will never be supported?

 

[cyberjock]

You are always welcome to submit bug reports. But more than likely the issue is with FreeBSD and so the ticket won't really fix anything, it'll just be open forever. There's lots of tickets in the FreeNAS system that have been created that will never be closed because its not a FreeNAS issue, but a FreeBSD one.

Also, since the intent of FreeNAS is to run it in bare metal configurations, support for ESXi is typically not something that the developers work on. But if you provide your own patch code to make it work with ESXi(or some other virtualization options) they'll implement it without much fuss.

The issue with ESXi not working is not an issue of "it not being supported". The issue is that FreeNAS (and especially ZFS) both are designed with the expectation that you will use it with bare metal and nothing else. Trying to use it with ESXi and RDM breaks those expectations, so all hell breaks loose. There is no 'redesigning' ZFS to make it work with non-bare metal. If you want that, what you are really looking for is nothing more than your standard desktop file system.

This is precisely why nobody with any VM knowledge has shown up in this thread. They KNOW that you're trying to put a square peg in a round hold when you want to do FreeNAS in a VM, especially with RDM. You sure can shave off the edges and make it fit. But someday when you need it to be a square you'll be disappointed to find that it isn't a square and no matter what you do it will never behave the same.

And look what happened when I tried to say that it just isn't smart. I was dismissed with a bad attitude, lack of knowledge, etc. They know they'll get the exact same treatment. Hence no posting. For some people, there is no fixing something that they are determine that should, can and will work and work well for them. Regardless of the overwhelming body of evidence that says you are wrong. The only reason I posted was that I was hoping to provide some knowledge that was more than "just don't do that.. its stupid". But instead I got an attitude over it, hence I chose to leave the thread.

And as much as I'd like ESXi to work with ZFS, I know it won't be fixed in this thread. And I know this won't be the last thread to try to prove that ZFS can and does work well with ESXi and RDM. The issue is a technical one, and one that nobody is trying to fix, because its not broken.

 

[VTecheira]

Thanks for the input Cyberjock. I don't think I will bother creating a bug; because as you said it's a FreeBSD 9.1.x and not a FreeNAS 9.1 issue. I'll have to pursue another option until then.

At the risk of derailing the thread, I'd like to talk about you being dismissed. I've been on the other side of the forum post so I feel compelled to comply.

First and foremost, your input was not being dismissed. I, for one, certainly wasn't; and I doubt if Flynn was either. Frankly most of what I know of FreeNAS and ZFS have been from the PPT you put together and your forum posts. So if I've not said it: thank you, your words and your work are appreciated.

Second, I've had some very costly and painful data losses in past so I'm acutely aware of the wisdom in your words and certainly don't want to relive the experience. That said, I think your first post didn't really help much either. It just said don't go that way there's risks and it's not the recommended approach. It didn't really take into account the fact that for Flynn, myself and probably a few others. The (risky and unrecommended) ESXi/RDM approach (and RAIDZ1) is probably our only feasible option at the time. And for my money a sub-optimal solution beats no solution at all. Now granted reading posts at 3am after wrestling with a tech issue probably didn't help on my end either.

Third, if you last post had been your first post I think this could have all gone quite differently.

FreeNAS definitely has the best interface of all the other options I've seen (and I so want to work with the new plug in jail). I guess that means I have to save up the scratch for a dedicated box. :(

Well that's enough feelings talk for a bunch of tech folks on a tech forum. Looking forward to being back on FreeNAS with bare metal. Thanks again for all the help. And your PPT should be required reading for all who dare to build a ZFS box.

P.S. For those who are interested, I've found OmniOS with Napp-It to be fairly capable in the interim. The UI is a little less friendly, but it works quite well.
P.P.S. I enable IOMMU on my board/bios but that didn't seem to have any effect on the original problem either.

 

[cyberjock]

Unfortunately, if I had made my last post my first my comments would have been dismissed. Some people just need to go through the paces of realizing it doesn't work before they accept the answer being provided to them already. In the Navy I was a nuclear reactor operator. We had a joke relating to problems and solutions. The joke was "knows answer when given". This was because too many people will ignore the answer even when it was blatantly obvious in the problem. Why? Because they were convinced that they could do it where everyone else had failed. This thread is no different. :P

I won't lie, I experimented with FreeNAS in a VM with the assistance of our ESXi expert. I have a thread here somewhere that somewhat documents my issues. After 8 long days or so of experimenting we gave up. It just wouldn't work in a configuration that was reliable without VT-d supported, enabled, and compatible hardware.

What happened when I bought the right hardware 2 months ago(and spent $700)? It worked the first time I booted it. Yes, I AM running FreeNAS as a VM in ESXi. And I followed every single recommendation that is recommended and not recommended in the forum. I also have liberal backups of my data just in case things go badly. But it hasn't been without problems. Right now I have an "Unknown: out of memory (5880)" error racking up in my ESXi logs every few hours with no explanation. And every so often all of VMs lose network activity and I can't shutdown ESXi after that, even if FreeNAS is the only VM running and with 10GB of RAM free. Tried to find the issue but I've given up. Bare metal config is in my future in the next few weeks when I have time to go that route.

Money is power, especially in computers. And some people aren't willing to accept that their hardware just won't do what they want, no matter how much they try. If you can't afford to spend the money on hardware you have 2 options; wait until you can afford it or experiment with it until you fail. Nobody wants to hear the words "wait", so this thread was born(just like all the others like this one from days gone by). Windows users look at it from the perspective of "it may be slow.. but it'll work". That's so far from the truth outside of Windows. Kernel Panics result, which isn't a "working" system, and can cause data loss. In short, the universe won't make your configuration magically work because you can't afford the right hardware. If you can't afford it then that's life.

But you can bet that some of the more experienced people have one or two names on their ignore list though thanks to this thread. With the way this forum goes, many of us volunteers(I'm one of them) just don't want to spend time dealing with issues from people that want validation that their ill-conceived plan is "recommended" and not actually wanting advice on how to do something right, or at least how to not do it wrong. We have so many people that show up, try to make something work that's been discussed until us mods feel nauseous, try to make it work, argue with us, and finally leave. Many of us have better things to do than deal with those people. I've probably dropped 2 hours just searching and writing for this thread? And for what benefit? I'm sure in the next month another thread just like this one will come about with a few more new people that are determined to get ESXi to work.

Here's a hint: No matter what your problem, if you aren't following some of the well intentioned recommendations of the forum, don't expect anyone to show up and help. It's one thing when you are trying to follow recommendations but had plenty of evidence that it should work(for instance, realtek versus Intel NIC), but blatantly ignoring the threads that were created only to save your data, expect no response from anyone with any knowledge in your thread. They've already told you what to do right, you chose to not listen. And us mods/experienced users really don't want to do one-on-one setups with every user that wants ESXi. We're not getting paid for one-on-one support. And if we were, we'd have told you to buy VT-d compatible hardware. Let's face it, ESXi can be an amazing tool and it can be a nightmare. Nobody wants to admit the nightmare but nobody will deny its an amazing tool either.

 

[pbucher]

This is precisely why nobody with any VM knowledge has shown up in this thread. They KNOW that you're trying to put a square peg in a round hold when you want to do FreeNAS in a VM, especially with RDM. You sure can shave off the edges and make it fit. But someday when you need it to be a square you'll be disappointed to find that it isn't a square and no matter what you do it will never behave the same.

Ok I'm a little slow but you jingled my chain now.....;)

Ok it sounds like Fln & VT have a fairly good idea of what they are doing. The really important thing to know is make sure you are using physical RDMs and not virtual ones. Virtual is indeed the kiss of death. Physical ones are fine and there is no reason to fear for your data if your drives are passing in unique serial #s which I saw someone verify that they where for at least one of the installs. The only real danger is vmware might remove or break the function in 6.0 or beyond since it's not exactly a supported method in their eyes, RDM was designed for passing though LUNs for fiber channel SANs, see below for more comments.

Also it looks like the root cause has been tracked down to a known FreeBSD bug. I'd suggest whoever wants to champion this issue go ahead and register at http://support.freenas.org and create a ticket with the important details posted in this thread. Several of the ix guys are also FreeBSD developers and can possibly poke the someone who is responsible for the broken code. I'd then monitor FreeBSD and watch for the bug fix, it would be nice if they'd fix it before 9.2 ships(which is very soon). If you can find when they patch FreeBSD and what the MFC # is for the patch you can probably then get the FreeNAS guys to pull it in for the next FreeNAS release. In the mean time stick to the 8.3.x series until this gets straightened out. Also please use LSI SCSI or SAS(I'd probably try SAS though SCSI might be a safer choice) has your controller type, buslogic might work but that's also looking for trouble down the road.

It's been well hashed out now that RDM isn't the best way to fly but some days it is the only way to fly. That said I've successfully run a production server using RDM physical disks(it just holds my backups so I'm not loosing sleep at night - but the backs do put some real stress on the VM & FreeNas during the backup session), it wasn't my 1st choice but I had no other choice. The server had a adaptec RAID controller that I could pass through but it had issues and it was not in my control to replace it with a desired LSI card. Using an adaptec controller with FreeBSD is probably worse then RDM pass through if you love your data(if you don't believe me read the source code for the adaptec controller). Eventually I was able to build myself an adaptec driver from source the worked ok and I switched from RDM to using that, just because I feared someday vmware would nuke RDM, that and I like being able to bring up my ESXi boxes from a fresh install without spending and bunch of time in the cli building RDM files to bring my SAN online.

I'd also check and make sure you can get some smart reports from the disks in RDM mode, the inability to monitor the hard drives is what gets folks up in arms about loosing data. That and way too many noobs make virtual RDMs which are simply bad and then when they are over 2TB in side the wheels come off the bus and it tumbles over the cliff all too quickly......

-Paul

 

[pbucher]

I just saw a posting about FreeBSD 9.2RC2 fixing some CAM errors for a physical server that where present in 9.1RC1, on the wild chance it could be the same problem see if FreeBSD 9.2RC2 allows you to do a zfs import of your pool. If it does I'd add that to the bug report and maybe they will sync up to the current FreeBSD code base.

 

[FlynnVT]

Thanks Paul! I'm not sure if FreeBSD-9.2-RC2 is available yet, but I just tried the latest 9.2 snapshot (PRERELEASE-amd64-20130811-r254222).

Unfortunately it gives the same result: discs are enumerated, model and serial # read OK but with a 0 byte media and 0 heads/sectors/sectorsize. Other symptoms are the same, e.g. "dd if=<dsk> of=/dev/null" gives "Device not configured". Hopefully, that CAM change commit you saw simply didn't make it to PR-20130811 and the 9.2 kernel will still fix this. I'll try again soon...

Interestingly - but not that it'll have any actual effect, changing the guest adapter from LSI Parallel to SAS makes the indicated transfer speed jump from 6.6 to 300 MB/s on both FreeBSD 8 and 9.

Finally, the BusLogic adapter isn't supported for 64-bit guests; giving an error message when attempting to start the machine. That rules out the second flawed workaround. 8.3.1 it is for now...