I may or may not get this 100% correct, so forgive me if later I find out I'm wrong..
1. A bug in ctl was identified causing the crashes mentioned in this thread. In particular, ctl was optimizing commands that were invalid commands, eventually leading to a crash. Of course we didn't know this until recently.
2. (I'm not sure of the specifics) but at some point this bug was introduced. I'm guessing it has happened in the last 60 days or so because we've only become aware of it recently. It was also hard to narrow down because of specifics to the customer's design that I'm not going to discuss. Suffice it to say, if we had seen this in FreeNAS first it may have been a faster turnaround to fix the bug, which leads me to #3.
3. I've been pretty busy lately, so I've been slacking on staying up on the forums. I have 18 pages of historical threads to catch up on. :/
4. Since nobody from the FreeNAS community created a bug ticket, when a TrueNAS support ticket was created for a TrueNAS user, the support team submitted a bug ticket. This went into the TrueNAS bug ticket system which is internal to iXsystems.
4. Customer that had the issues worked with us and we eventually issued a fix. That fix went into TrueOS.
TrueOS is basically "FreeBSD" with custom stuff added, removed, or changed that makes the basis for FreeNAS and TrueNAS. Its based on FreeBSD, but has it's own fork specifically for things like FreeNAS and TrueNAS.
5. Since the ticket was an internal ticket, and since FreeNAS shares the code with TrueOS, the fix was included in the most recent FreeNAS build. But since the TrueNAS ticket is internal and no FreeNAS equivalent was ever made, there doesn't appear to be a changelog for the bug (or the fix) but it is most certainly there. It's just a matter of tracking everything back appropriately.
At the end of the day there's a few things to keep in mind:
1. If you are getting crashes, and you are convinced its not a hardware failure, put in a bug ticket and attach a debug file. Let the developers figure out if you have a poor configuration, your hardware is bad, or if this is a genuine problem and needs fixing. Panics should *not* typically be happening if you are sticking to recommended gear and taking all of the advice given by the experienced users here in the forums to heart. In this case nobody seems to have put in a bug ticket, so the FreeNAS community didn't really head this one off, which meant it too longer to identify because TrueNAS users are typically more interested in uptime until we can get a definitive reproduction case, etc. If you are a big customer you may have HA, which means you may have crashes every day and you may not know it because the workloads typically failover seamlessly. So unless you look at the uptime (which I think is what the TrueNAS user noticed) then you may not know. (yay for TrueNAS HA!)
What made this worse was that we couldn't reproduce this in a lab environment, and while some people saw the issue regularly, the vast majority never had a problem. I personally have been using iSCSI for more than a year, but never hit this bug. And with all of the weird and nasty things I've done to my system on the software side, and doing "all that crap I tell you end-users to never do to your production systems" I'd have hoped to have accidentally hit the problem.
2. Internal tickets for TrueNAS are basically 'hidden' because they often contain customer data. Not that we're trying to hide big juicy bugs and major security holes from the world, but we need a place to do things that may include attaching things that are sensitive to our paying customers. Often the big juicy bugs are first identified by the FreeNAS community, so the FreeNAS ticket tracks the bug to eventual completion. It just didn't for this case because of #4 above.
3. If you are having this particular problem, it is in your best interest to upgrade to the latest STABLE build. If you are still experiencing this issue afterward, iXsystems will almost certainly want to know.
Likewise in the future if you are having panics, it's a good idea to put in a bug ticket, especially if several people can confirm the same cause. That's almost certainly a sign that a bug exists and needs to be squashed ASAP.
Right now we've applied the fix for the TrueNAS customer that had the problem, and they've not had the problem since. Likewise it sounds like nobody that uses FreeNAS has had the problem if they have upgraded to the latest build of FreeNAS. So I think the patch that intended to fixed the issue has been successful and we can claim victory over the panics. (yay!)
Remember, nothing I say here in the forums or IRC is "official statements from iXsystems" so ymmv.
