SweetAndLow, First Thanks for all the Help. (thank you, Thank You)
I was finally able to identify the problem and have ZFS being resilvered with new 4TB drive that was used to replace. Just an FYI what I discovered and how I discovered it.
When I got hope I decided my first step was to map out each drive and the slot it was in and compared serial numbers to those that I had done a screen capture. What I discovered was:
In Tower 1: (This was the tower that was the original tower put on line with 10 4 TB drives.)
- The last drive in that tower was not showing up in the serial #'s. (This ended up being the drive that was causing all the problems)
In Tower 2: This tower was only to have only the top 5 drives online which were all 4 TB. While mapping it out I discovered that in the bottom row of 5 drives a 2TB drive had been pushed into bay completely. This should not have been in and Freenas for some reason (possiblely a bug) picked it up in place of the drive that was not showing up in tower 1 as part of the pool. Note: I have entire bottom 5 rows of this tower filled with drives but they are not inserted completely in primarly to muffle out the fan noise. I recalled that on 1/15 I was working under the desk where the towers are located moving stuff and its very possible that it may have been pushed in accidetnly as I was moving things around. (Not sure how the 2TB drive would have affected the drive on Tower 1.)
I started with removing the 2TB drive and the last drive in Tower 1. This solved the boot issue and the server booted up with entire pool online but degraded.
I then tried a 2nd reboot with the last Drive from Tower 1 reinserted. When I tried booting up again the error and crash occured.
I then replaced the last drive in tower 1 with a new drive and tried a boot once again. This solved the issue. Server booted up and loaded up the degraded pool. I then used the replace command to resilver pool with new drive.
All is well now and looks like I will recover all my data once the resilvering is complete.