Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Bootlooping only when storage enclosure attached

Joined
Apr 25, 2019
Messages
6
Thanks
0
#1
Hey guys,

I just had the board fail in my stand alone system for freenas and now for a couple months until I can upgrade that I need to move Freenas to a VM with Passthrough. I am on ESXI 6.7 and running freenas 11.2u4. whenever I power up the VM it start the boot process and goes until I get error "(probe24:mps0:0:33:0): SCSI Status: Busy" and then immediately reboots. My setup and specs are below. I have the EXP3000 Plugged in to the LSI SAS9211-8i with using a single cable to the 'in' port on one of the ESM modules. The vm will keeping rebooting at the same point over and over. If I unplug the Sas cable connecting the enclosure the vm will boot fine. If I wait until Freenas finishes booting I can plug the SAS cable back in and then import the pool and everything runs fine until I need to reboot again. I attached a photo of the error which more details. Any thoughts here? Side note all disks are reported as healthy within freenas when I plug in the enclosure after letting it boot up. I'd really love to figure this out as I remote in to my lab to find myself rebooting a lot(self inflicted) and can't always unplug sas cable during boot.

VM Specs:
6vCPUS
12 GB Memory
LSI SAS9211-8i directly passthrough to VM

Specs:
IBM X3650 M3:
- 2x Xeon X5650 2.67Ghz 6c/12t
- 72GB DDR3 Reg ECC memory
- LSI MegaRaid M1015
- 3x SFF 146GB 10K SAS drives (RAID5, ESX Boot drive)
- LSI SAS9211-8I (flashed to IT mode)
- IBM EXP3000 SAS storage enclosure:
-12x 1TB LFF IBM 7.2K SAS drives
 

Attachments

Chris Moore

Super Moderator
Moderator
Joined
May 2, 2015
Messages
9,350
Thanks
2,984
#2
How are you passing the controller in to the VM? I know there are several folks here running FreeNAS in a VM, but it is important how the hardware is passed in.
You might want to take a look at some of these threads to see if you can find any hints:

Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://www.ixsystems.com/community...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

and

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.
https://www.ixsystems.com/community...ide-to-not-completely-losing-your-data.12714/

Virtually FreeNAS ... an alternative for those seeking virtualization
https://www.ixsystems.com/community...ative-for-those-seeking-virtualization.26095/

FreeNAS 9.10 on VMware ESXi 6.0 Guide
https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/
 
Joined
Apr 25, 2019
Messages
6
Thanks
0
#3
Hi Chris,

Thanks for the reply. I had reviewed 2 of those links before I started with the freenas install on VM. I have enabled Passthrough under the hardware section of the host as shown in the first image below. I then have the VM configured as in the second screenshot.

2019-08-11 14_39_21.png

1565549314114.png
 
Joined
Apr 25, 2019
Messages
6
Thanks
0
#4
Hello,

so I did some additional testing and in an attempt to narrow down the issue I booted the host to freenas installer and installed a flash drive. The install goes fine and will complete the boot process without any issue if the EXP3000 is not connected. If I let freenas boot fully and then plug in the enclosure all disks in the exclosure are detected and run without issue until a reboot. At which point the same error is displayed. The only difference with this is that the system does just keep rebooting like the VM. This stays at a KDB panic. New screenshot of error below. So this does not appear to have anything to do with freenas running as a VM in ESXi. Hoping this changes the direction and someone has a thought or suggestion to try. I have been searching various sites and not turning up any results. The closest thread I can find is link below but a resolution was never found. The errors are almost identical though. The last thought was firmware for the enclosure but I cannot download the files from IBM without a support contract. I am still trying to find the files through an alternative means.

https://www.ixsystems.com/community...-and-9200-8e-controller-to-exp3000-das.50595/
 

Attachments

Joined
Apr 25, 2019
Messages
6
Thanks
0
#5
So another thought just occurred to me while re-reading some stickies and posts for something I might have missed. In the "Don't be afraid to be SAS-sy" post in the cabling seciton it indicates that internal connectors can be transformed into external by using an adapter plate. The current setup has the EXP3000 enclosure connected using SFF-8088 to SFF-8088 1M External Cable connected to a Supermicro Backplane Cable with 1-Port Internal Cascading Cable (CBL-0167L) which takes the SFF-8088 to SFF-8087 which plugs directly into the LSI SAS9211-8i.

Is this not how I should be doing it? Should I change it out for a SFF-8088 to SFF-8087 cable?

https://www.amazon.com/gp/product/B00S7KTXW6/ref=ppx_yo_dt_b_asin_title_o02_s00?ie=UTF8&psc=1
https://www.amazon.com/gp/product/B000MBOR56/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1
 
Joined
Apr 25, 2019
Messages
6
Thanks
0
#6
Updating this at this point just for completeness sake. I am at this time unable to get the EXP3000 to work with freenas 11.2U5 or any earlier version.

I had bought a new SFF-8088 to SFF-8087 cable and get the same results, cam status busy timeouts and then a KDB panic. I pulled all the drives form the encolure and there is no change either. After many many hours of testing options was able to get a couple times of plugging in the enclosure after fully booting freenas(no longer in ESX and is installed to a 16gb usb drive on the server) where freenas detected the enclosure and all 12 drives. I also tested the enclosure with an entirely physical server and get the same issue. Rarely when freenas does detect the enclosure without causing KDB panic I was able to identify the device that camstatus returns the errors for and then crashes. It is SES0 which to my understanding in the ESM module in the enclosure. I have dual esm in this enclosure and I have tried all the combinations on 1 at a time and in each slot with no change. When reviewing the firmware update change history(see below) it specifically calls out panic issue in many of the versions. The real kicker here is that IBM requires a support contract to download the firmware(consider the latest release of the firmware is from 2012 it blows my mind that this is still the case.)

IBM System Storage™ DS3000 EXP3000 ESM Firmware Update package
Version 1.9A


NOTE TO SERVICE – Reference RETAIN # N/A


Last Update: 06/14/2012

June 14, 2012
- Fix LSIP200290974 ESM panic during gathering state captures as part of CASD
collecting
- Fix LSIP200135690 Access to drives interrupted after 497 days of continuous use.
- Fix 181987 (RP #WIC~178283) 6844 ESM PWR Supply intermittently fluctuation
- Fix CR 148536 (c444788) RTP_EX12 Discovery Error 0x10 with expander version
1.16.e57


November 5, 2008 - EXP3000 ESM v01.96
- Fixed an issue where the SAS expander would encounter a PHY discovery
error and cause the ESM to panic.

April 21, 2008 - EXP3000 ESM v01.93
- Fixed a rarely seen issue were, in a dual ESM system, one ESM would panic
and reboot. During the reboot, the other ESM would panic and also


The only options I see are finding another download of the firmware which I have spent days and days on trying to locate or purchase a replacement esm module and hope the firmware is updated to the newest. Given that I can't confirm 100% the updated firmware will even help me I am feeling like I might just get rid of the exp3000 and look for something a bit newer and without the support contract bullcrap. I leave this open to any suggestions if anyone can think of anything or if you anyone wants to recommend a new enclosure would be great a bonus if I use the 12 1tb sas drives I have in this enclosure but not a requirement.

To those that read through all of this thank you. It was a fun ride and I learned a lot through this process but too demotivated to keep pouring money into uncertainties with this enclosure. I am in the meantime using the 2.5 inch bays in the x3650 for freenas mainly filled with SSDs just for storage to mess around with. The goal for the enclosure was longer term storage but I'll shelf that idea for the moment and get back to some other projects I was working on.
 
Top