Thanks for the reply... this is definitely a tough one to troubleshoot and I appreciate the help I know this post was slim on details, I was seeing if someone recognized the panic message.
All the details I can remember:
The drives are all in one pool, 2 vdevs. RAIDZ2. Drives are all WD green drives and a few Seagate drives. I had memtested my old system (at build and when I started troubleshooting this issue) and everything was fine, but I haven't tested the new system yet. Just got everything installed a few days ago and since the problems persisted, I figured that the hardware wasn't the problem. I'll stress test them after I figure this out or have to rebuild the pool.
If I disconnect all of the drives the server is stable. The odd part is that it will sometimes boot and import normally and then randomly crash usually within an hour. There is no real pattern to it that I can figure out.
I can force an import into readonly mode using #zpool import -o readonly=on -f <poolname> but now that I'm going over it all again in my head I don't think I've left it in readonly mode for longer than a few minutes while troubleshooting so I don't know if that crashes the system. I will try that today while trying to pull some media files and report back.
Thanks for the links, I'll read them, run the SMART tests this weekend and report back. I had them scheduled to run regularly, but reinstalled FreeNAS trying to fix this problem a few weeks ago when I started troubleshooting and haven't set them up yet. Only hard drive issue I'm having is pretty recent (about a week ago). I started getting a critical warning that a drive had uncorrectable and unreadable sectors, but I haven't had a chance to replace it yet. I'm not sure how a resliver would go at this point, and if I disconnect the failing drive I still have the same issues. Not sure if this is related to the error, or just a failing drive making my life difficult at the moment.
The data is mostly photo file backups and my media collection. It isn't anything that I'd be devastated at losing, but it is about 6TB of data and I'd really like to exhaust all possibilities before destroying the pool since I know the data is still there and I can access it for limited periods of time. I have backups of everything except some of the media files.
As for changes,
I thought this problem only started a few weeks ago but looking through my emails to get you a better timeline, I am seeing that it was actually rebooting as early as 2 months ago, possibly before, but only once a week. I was on 11.1 at the time and I guess that I thought that whatever it was would get patched out. Damn, thought it was just a random hardware issue this whole time but now I guess it could have been a bad upgrade as well. Would a bad upgrade cause the problem get worse weeks later? This pool was in my older system at that time, i5, 8GB non-ECC ram, LGA1155 EVGA motherboard (don't remember the exact model number), with a SATA expansion card. I tried a downgrade during all of this troubleshooting to v.11.0 U4 and the pool would actually boot on that version, but didn't put 2 and 2 together that the issue may have started with 11.1. It actually ran great for a week or so until the issues came back.
I thought I had a hardware issue so I replaced the power supply. Didn't work. I then thought it might be a SATA controller on the motherboard since the error only occurred when using the disks. I figured it was a good opportunity to replace the older hardware (and non-ECC RAM) and replaced the Mobo / processor (i3) / ram a few days ago. The issue has persisted. The only hardware I haven't replaced are any of the hard drives or cables.
Thanks again for the assistance, it is greatly appreciated.
