Kernel panic when upgrading TrueNAS 12 to 13

[Davvo]

You could try jerking the os drive off, putting in another machine and installing 13 there, then putting back to your nas.
If the issue is only with the installer this should skip it.

 

[Agent92]

Tried to remove the HBA-card, network card and three RAM sticks and boot it again, no difference. Tried a different RAM-stick, no difference. Moved to a different RAM-slot no difference. Have not yet tried to reseat the cables or CPU since that is a bigger job but will try it.

Seems like it flip flops between complaining about core 6 or 7, not sure if that is relevant.

Tried to install TrueNAS 13 on a different machine and move the drive over, same issue.

 

[Agent92]

Just to try everything I tried the TrueNAS 12 installer and it worked fine.

 

[Agent92]

I'm also thinking possibly bent CPU pin but it feels like that wouldn't affect just 13 so consistently. Will have to get a magnifying glass with lamp first to inspect that though so will take a break here until I can get that unless someone has some other ideas.

 

[joeschmuck]

Check the BIOS revision on your motherboard. The current version is: Version 3802 and if your BIOS is not current, I'd recommend that you update it and cross your fingers it works.

And of course I'm going to throw this out there, This is not a supported motherboard for TrueNAS, it does not meet the system specs, No ECC RAM. But that is fine if you accept it.

 

[joeschmuck]

I'm also thinking possibly bent CPU pin but it feels like that wouldn't affect just 13 so consistently.

I highly doubt this would be the cause. You are apt to induce a bent pin if your remove it and reinstall it.

As for thermal paste... I found out that placing a dab of it in the center of the CPU vice spreading it is actually a better way, it causes no air pockets and might solve the few hotter cores you have. But you may do this already. I've always preferred to spread it but it explains why I may have had a few hotter cores in the past.

 

[Agent92]

Check the BIOS revision on your motherboard.

Already got the latest version.

This is not a supported motherboard for TrueNAS, it does not meet the system specs, No ECC RAM. But that is fine if you accept it.

I know, I accept that, can't really motivate ECC for a home build until DDR5 starts to trickle down from my main machine

As for thermal paste... I found out that placing a dab of it in the center of the CPU vice spreading it is actually a better way, it causes no air pockets and might solve the few hotter cores you have.

Yeah I have found this as well, in this case I'm using Kryonaut Extreme though which is a bit special when it comes to the application. I don't really see how temps would be the issue when it works fine with 12 and prime95 doesn't go over 65c.

Thank all for the help so far, will try to reseat the CPU and look for bent pins (even if it's unlikely), if I find nothing I will file a bug report and see if the devs have any input.

 

[Davvo]

Welp if the reseating doesn't work only a dev can help you. Either that or a change of CPU.

Also it will be a while until we see ECC DDR5 on server motherboards. A long while if we talk about used ones.

 

[joeschmuck]

Yeah I have found this as well, in this case I'm using Kryonaut Extreme though which is a bit special when it comes to the application. I don't really see how temps would be the issue when it works fine with 12 and prime95 doesn't go over 65c.

I wasn't trying to imply that the error was the thermal paste application. Just some good to know stuff is all.

As for bent pins, guess you are suspecting the motherboard has a bent pin. I still find that hard to believe since all the other testing you performed worked fine. I did a little research and found that the error message is complaining of a hardware error.

One thing you could do is try to run FreeBSD 13.1 and see what happens. Maybe it will work fine. If it does, then I'd submit a bug report on TrueNAS.

I also assume you are trying to install TrueNAS 13.0-U3.1 (which is built from FreeBSD 13.1-Release-p2. And you did say you tried a clean install I think.

 

[Agent92]

guess you are suspecting the motherboard has a bent pin

Correct.

One thing you could do is try to run FreeBSD 13.1 and see what happens.

Good idea, I will try that.

And you did say you tried a clean install I think.

Never even got to the install part since the installer itself exits with the error.

 

[Agent92]

Very late update but I only now had the chance to test with different hardware and I can now say that the CPU is broken. So it's quite impressive that it still ran for almost a year and only crashing around 5 times.

 

[joeschmuck]

It's good you found the failing item. Did you pop the CPU into another motherboard or pop a new CPU into the current motherboard? I ask because it I very possible that a motherboard VRM could have harmed the CPU. Every tenth of a volt is a large change when you are dealing with the lithography of todays CPUs.

 

[Agent92]

I did both actually so result was very conclusive.