CreeMichaud
Cadet
- Joined
- Dec 21, 2020
- Messages
- 4
Hardware:
Supermicro X8DAL with 2x Intel x5660
64 GB ECC RAM
SAS9207-8i (LSISAS2308) BIOS: 7.39.02.00 FW: 20.00.07.00
Environment:
TrueNAS-12.0-U1 running as VM (this is a non-prod homelab)
Hosted on XCP-ng 8.2.0 (latest patches)
SAS HBA is PCI-Passthrough to guest OS (TrueNAS)
Symptom:
VM reboot gets stuck at "mp0: reinitializing" unless I set "verbose" mode from boot menu (opt 6) or I reboot entire XCP-ng host
Background:
This hardware has been running without issue as a FreeNAS 11.3-U4 guest on XCP-ng 8.1.0. I've completed multiple in place upgrades previously without issue.
No hardware changes, no BIOS changes, no Firmware changes in the environment. Only change in environment was migration to TrueNAS and apply XCP-NG patches.
Installation and first boot to TrueNAS occurred without incident. Attached disks and imported pool without issue. When restoring TrueNAS config, the reboot gets stuck.
Terminal won't get passed "mp0: reinitializing"
VM Boot never progresses no matter how long I wait. Restarting the VM yields the same symptom. If I Reboot XCP-ng host it resolves the issue. A subsequent VM reboot gets stuck again. Strangely, one work around I stumbled upon was to set "verbose" option from boot menu. Most of the time I can get it to boot without rebooting the XCP-ng host. Subsequent VM reboots get stuck again.
Once TrueNAS actually boots up, the system appears to be stable enough to complete a scrub on 10TB of encrypted raidz2 in 10hrs with no system activity. I have not found any suspicious log entries related to disk/HBA subsystems in either VM or HOST logs
I have reinstalled a fresh copy of TrueNAS instead of an in place upgrade. No change in behavior
I have also tested/found the same behavior on FreeNAS 11.3-U5 and TrueNAS 12.0-RC1
I have not tried to roll back XCP-ng to 8.1.0 or to try bare metal TrueNAS with same hardware config. Both of these options are a large investment in time. I unfortunately don't have a spare server/SAS HBA. A rollback would require a complete wipe / restore of all VMs
I find it odd the "verbose" boot option seems to affect the symptom. I'm not sure how to leverage this data point to find the solution.
Does anyone have a suggestion I can try before rolling it all back?
Any help/advise is appreciated, thanks in advance.
Supermicro X8DAL with 2x Intel x5660
64 GB ECC RAM
SAS9207-8i (LSISAS2308) BIOS: 7.39.02.00 FW: 20.00.07.00
Environment:
TrueNAS-12.0-U1 running as VM (this is a non-prod homelab)
Hosted on XCP-ng 8.2.0 (latest patches)
SAS HBA is PCI-Passthrough to guest OS (TrueNAS)
Symptom:
VM reboot gets stuck at "mp0: reinitializing" unless I set "verbose" mode from boot menu (opt 6) or I reboot entire XCP-ng host
Background:
This hardware has been running without issue as a FreeNAS 11.3-U4 guest on XCP-ng 8.1.0. I've completed multiple in place upgrades previously without issue.
No hardware changes, no BIOS changes, no Firmware changes in the environment. Only change in environment was migration to TrueNAS and apply XCP-NG patches.
Installation and first boot to TrueNAS occurred without incident. Attached disks and imported pool without issue. When restoring TrueNAS config, the reboot gets stuck.
Terminal won't get passed "mp0: reinitializing"
VM Boot never progresses no matter how long I wait. Restarting the VM yields the same symptom. If I Reboot XCP-ng host it resolves the issue. A subsequent VM reboot gets stuck again. Strangely, one work around I stumbled upon was to set "verbose" option from boot menu. Most of the time I can get it to boot without rebooting the XCP-ng host. Subsequent VM reboots get stuck again.
Once TrueNAS actually boots up, the system appears to be stable enough to complete a scrub on 10TB of encrypted raidz2 in 10hrs with no system activity. I have not found any suspicious log entries related to disk/HBA subsystems in either VM or HOST logs
I have reinstalled a fresh copy of TrueNAS instead of an in place upgrade. No change in behavior
I have also tested/found the same behavior on FreeNAS 11.3-U5 and TrueNAS 12.0-RC1
I have not tried to roll back XCP-ng to 8.1.0 or to try bare metal TrueNAS with same hardware config. Both of these options are a large investment in time. I unfortunately don't have a spare server/SAS HBA. A rollback would require a complete wipe / restore of all VMs
I find it odd the "verbose" boot option seems to affect the symptom. I'm not sure how to leverage this data point to find the solution.
Does anyone have a suggestion I can try before rolling it all back?
Any help/advise is appreciated, thanks in advance.