Bootlooping only when storage enclosure attached

SatchaDahmi

Cadet
Joined
Apr 25, 2019
Messages
7
Hey guys,

I just had the board fail in my stand alone system for freenas and now for a couple months until I can upgrade that I need to move Freenas to a VM with Passthrough. I am on ESXI 6.7 and running freenas 11.2u4. whenever I power up the VM it start the boot process and goes until I get error "(probe24:mps0:0:33:0): SCSI Status: Busy" and then immediately reboots. My setup and specs are below. I have the EXP3000 Plugged in to the LSI SAS9211-8i with using a single cable to the 'in' port on one of the ESM modules. The vm will keeping rebooting at the same point over and over. If I unplug the Sas cable connecting the enclosure the vm will boot fine. If I wait until Freenas finishes booting I can plug the SAS cable back in and then import the pool and everything runs fine until I need to reboot again. I attached a photo of the error which more details. Any thoughts here? Side note all disks are reported as healthy within freenas when I plug in the enclosure after letting it boot up. I'd really love to figure this out as I remote in to my lab to find myself rebooting a lot(self inflicted) and can't always unplug sas cable during boot.

VM Specs:
6vCPUS
12 GB Memory
LSI SAS9211-8i directly passthrough to VM

Specs:
IBM X3650 M3:
- 2x Xeon X5650 2.67Ghz 6c/12t
- 72GB DDR3 Reg ECC memory
- LSI MegaRaid M1015
- 3x SFF 146GB 10K SAS drives (RAID5, ESX Boot drive)
- LSI SAS9211-8I (flashed to IT mode)
- IBM EXP3000 SAS storage enclosure:
-12x 1TB LFF IBM 7.2K SAS drives
 

Attachments

  • IMG_20190810_232700.jpg
    IMG_20190810_232700.jpg
    212.3 KB · Views: 286

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
How are you passing the controller in to the VM? I know there are several folks here running FreeNAS in a VM, but it is important how the hardware is passed in.
You might want to take a look at some of these threads to see if you can find any hints:

Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://www.ixsystems.com/community...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

and

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.
https://www.ixsystems.com/community...ide-to-not-completely-losing-your-data.12714/

Virtually FreeNAS ... an alternative for those seeking virtualization
https://www.ixsystems.com/community...ative-for-those-seeking-virtualization.26095/

FreeNAS 9.10 on VMware ESXi 6.0 Guide
https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/
 

SatchaDahmi

Cadet
Joined
Apr 25, 2019
Messages
7
Hi Chris,

Thanks for the reply. I had reviewed 2 of those links before I started with the freenas install on VM. I have enabled Passthrough under the hardware section of the host as shown in the first image below. I then have the VM configured as in the second screenshot.

2019-08-11 14_39_21.png

1565549314114.png
 

SatchaDahmi

Cadet
Joined
Apr 25, 2019
Messages
7
Hello,

so I did some additional testing and in an attempt to narrow down the issue I booted the host to freenas installer and installed a flash drive. The install goes fine and will complete the boot process without any issue if the EXP3000 is not connected. If I let freenas boot fully and then plug in the enclosure all disks in the exclosure are detected and run without issue until a reboot. At which point the same error is displayed. The only difference with this is that the system does just keep rebooting like the VM. This stays at a KDB panic. New screenshot of error below. So this does not appear to have anything to do with freenas running as a VM in ESXi. Hoping this changes the direction and someone has a thought or suggestion to try. I have been searching various sites and not turning up any results. The closest thread I can find is link below but a resolution was never found. The errors are almost identical though. The last thought was firmware for the enclosure but I cannot download the files from IBM without a support contract. I am still trying to find the files through an alternative means.

https://www.ixsystems.com/community...-and-9200-8e-controller-to-exp3000-das.50595/
 

Attachments

  • IMG_20190811_231818.jpg
    IMG_20190811_231818.jpg
    440 KB · Views: 283

SatchaDahmi

Cadet
Joined
Apr 25, 2019
Messages
7
So another thought just occurred to me while re-reading some stickies and posts for something I might have missed. In the "Don't be afraid to be SAS-sy" post in the cabling seciton it indicates that internal connectors can be transformed into external by using an adapter plate. The current setup has the EXP3000 enclosure connected using SFF-8088 to SFF-8088 1M External Cable connected to a Supermicro Backplane Cable with 1-Port Internal Cascading Cable (CBL-0167L) which takes the SFF-8088 to SFF-8087 which plugs directly into the LSI SAS9211-8i.

Is this not how I should be doing it? Should I change it out for a SFF-8088 to SFF-8087 cable?

https://www.amazon.com/gp/product/B00S7KTXW6/ref=ppx_yo_dt_b_asin_title_o02_s00?ie=UTF8&psc=1
https://www.amazon.com/gp/product/B000MBOR56/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1
 

SatchaDahmi

Cadet
Joined
Apr 25, 2019
Messages
7
Updating this at this point just for completeness sake. I am at this time unable to get the EXP3000 to work with freenas 11.2U5 or any earlier version.

I had bought a new SFF-8088 to SFF-8087 cable and get the same results, cam status busy timeouts and then a KDB panic. I pulled all the drives form the encolure and there is no change either. After many many hours of testing options was able to get a couple times of plugging in the enclosure after fully booting freenas(no longer in ESX and is installed to a 16gb usb drive on the server) where freenas detected the enclosure and all 12 drives. I also tested the enclosure with an entirely physical server and get the same issue. Rarely when freenas does detect the enclosure without causing KDB panic I was able to identify the device that camstatus returns the errors for and then crashes. It is SES0 which to my understanding in the ESM module in the enclosure. I have dual esm in this enclosure and I have tried all the combinations on 1 at a time and in each slot with no change. When reviewing the firmware update change history(see below) it specifically calls out panic issue in many of the versions. The real kicker here is that IBM requires a support contract to download the firmware(consider the latest release of the firmware is from 2012 it blows my mind that this is still the case.)

IBM System Storage™ DS3000 EXP3000 ESM Firmware Update package
Version 1.9A


NOTE TO SERVICE – Reference RETAIN # N/A


Last Update: 06/14/2012

June 14, 2012
- Fix LSIP200290974 ESM panic during gathering state captures as part of CASD
collecting
- Fix LSIP200135690 Access to drives interrupted after 497 days of continuous use.
- Fix 181987 (RP #WIC~178283) 6844 ESM PWR Supply intermittently fluctuation
- Fix CR 148536 (c444788) RTP_EX12 Discovery Error 0x10 with expander version
1.16.e57


November 5, 2008 - EXP3000 ESM v01.96
- Fixed an issue where the SAS expander would encounter a PHY discovery
error and cause the ESM to panic.

April 21, 2008 - EXP3000 ESM v01.93
- Fixed a rarely seen issue were, in a dual ESM system, one ESM would panic
and reboot. During the reboot, the other ESM would panic and also



The only options I see are finding another download of the firmware which I have spent days and days on trying to locate or purchase a replacement esm module and hope the firmware is updated to the newest. Given that I can't confirm 100% the updated firmware will even help me I am feeling like I might just get rid of the exp3000 and look for something a bit newer and without the support contract bullshit. I leave this open to any suggestions if anyone can think of anything or if you anyone wants to recommend a new enclosure would be great a bonus if I use the 12 1tb sas drives I have in this enclosure but not a requirement.

To those that read through all of this thank you. It was a fun ride and I learned a lot through this process but too demotivated to keep pouring money into uncertainties with this enclosure. I am in the meantime using the 2.5 inch bays in the x3650 for freenas mainly filled with SSDs just for storage to mess around with. The goal for the enclosure was longer term storage but I'll shelf that idea for the moment and get back to some other projects I was working on.
 

SatchaDahmi

Cadet
Joined
Apr 25, 2019
Messages
7
Hey thanks so much. A big oversight on my part. I was having a hell of a time getting the update app to run from a usb. kept having driver problems with the ibm autorun that happens. Compound that with not a support ibm megaraid card I had to change tactics. I used the sg3_utils package with the sg_write_buffer command to load the firmware used a linux vm. Was able to flash it to both esm controllers and freenas no longer has any issues with booting or running with exclosure attached. The firmware was the definitely the issue the whole time.

I had actually took all the drives and were using them in a converted old desktop case as a dummy jbod enclosure with a sas expander. I moved them all back to the IBM enclosure and everything is working great. Good news is I have the other case empty and ready more drives when I need to add more space thanks!
 

netappppp

Cadet
Joined
Jul 9, 2021
Messages
1
I have the same issue, I don't suppose you could give me a walkthrough of how you flashed the firmware? Would be much appreciated! Thank you!


Hey thanks so much. A big oversight on my part. I was having a hell of a time getting the update app to run from a usb. kept having driver problems with the ibm autorun that happens. Compound that with not a support ibm megaraid card I had to change tactics. I used the sg3_utils package with the sg_write_buffer command to load the firmware used a linux vm. Was able to flash it to both esm controllers and freenas no longer has any issues with booting or running with exclosure attached. The firmware was the definitely the issue the whole time.

I had actually took all the drives and were using them in a converted old desktop case as a dummy jbod enclosure with a sas expander. I moved them all back to the IBM enclosure and everything is working great. Good news is I have the other case empty and ready more drives when I need to add more space thanks!
 
Top