SOLVED checksum fail while installing FreeNAS 11.3

[mistermanko]

It has been a long, rough ride but I think I found the culprit, more at the the end of the post.

Edit 19th of March:

After switching the mainboard which turned out not to be the solution, I switched to another cpu which in result was the final solution of all checksum errors. In the end, I had to buy a whole new set of hardware to find the error in the CPU. I have no idea if the CPU was having a hardware fault or if just FreeBSD had some compatibility issues with the 1230v3, I doubt it though. Because all hardware tests, be it memtest or benchmarking tools did not show any issues with the CPU it was impossible to confirm, but switching to a 1245v3 brought me peace. I could activate all cores and ran installs and updates which went all fine. Maybe I was really unfortunate, but it was expensive af to be honest to buy two sets of mobo, memory and mainboard just to find out the cpu had issues.



I decided to re-write this Post, as it did not provide enough information for experts to get a look at.
Further down might be a hint to a bug in FreeNAS 11.3, stay tuned!

What I intended to do?
I build a FreeNAS system with specs as follows:

PSU: be quiet! Pure Power 11 500W 80+ Gold
MOBO: Supermicro X10SLL-F
RAM: 16 GB ECC RAM Samsung (2x M391B5273DH0; 2x M391B5173QH0)
CPU: Intel Xeon E3-1230v3
BOOT-POOL: 2x WD Green SSD SATA 240GB big no-no as I've learned
STORAGE-POOL: 6x 4TB WD RED (RAIDZ2)
HBA: LSISAS2008 IT-mode (x-flashed Dell H310)

I tried to install FreeNAS 11.3 via a USB 3.0 thumb drive and failed miserably.

Where did I fail?

When booting the installer it stops at ChecksumFailException: ...11.3-RELEASE-....tgz has invalid checksum, see here:

What did I try to fix this?

• tried different USB-Sticks (2x 3.0, 1x 2.0)
• tried installing via Legacy and via UEFI
• tried installing on different boot devices (2 x WD Green 240G drives, newly bought; the USB sticks from above; a 320G 2.5 spinning rust)
• tried installing via IPMI virtual storage
• of course did I download and checksum verify the .iso several times over the last 3 days
• set the BIOS to defaults, re-arranged the boot device list, checked the manual for UEFI/Legacy/boot settings, verified that I had the latest FW/BIOS installed (Supermicro FW rev 01.35; BIOS ver 3.2)
• studied the gpart man page and re-formatted my thumb drives and disks many many times

All of that didn't get me any further, so
here's why I think it is a bug:

• Every possible boot device mentioned above, got installed successfully with a second system (MSI b450 gaming).
• Version 9.10.2-U6.iso installed successfully via one of the USB drives onto the 320G HDD on the Supermicro system.

I'm not alone:

• /u/cassato on reddit had the same problem and fixed it by de-attaching the USB 3.0 header on his board
• @cyberfly got the same error while updating from 11.2
• @JohnnyAlpha got the same error and @cfs decided to use CD-ROM out of frustation

So where do we go from here? File a bug report? Re-sell my hardware and go on with my life?

######

EDIT on 28th of February: After almost an entire month and two tickets in the bugtracker, I could confirm the failure is in my hardware.
It's either the CPU or the Motherboard or a combination of them both.
I could reproduce the issue successfully and I have now a reliable solution to prevent the issue.

Vladimir over at jira was helping me out to find that my CPU had trouble with cryptographic functions, which caused a whole bunch of weird issues on FreeNAS.
In the history of events I eventually could get a finished install of 11.3 up and running but was soon overwhelmed with frequent error messages which were all over the place.
The email-alert service was spamming me with nonsensical alerts, the GUI felt janky and here and there was stuff not functioning as expected.

So I started to turn-off CPU features in the BIOS and run the installer over and over again to check if it interferes with the checksum validation of the base-os.tgz.
First I suspected AES-IN to misbehave but that wasn't the case. I went on with VT-x, then HT, then power saving and so on. I ended up disabling all CPU features. Still, the installer wouldn't perform a complete install of FN reliably. As mentioned further up, it sometimes finished, sometimes it did not. So all whats left was disabling CPU-cores.
...well and what can I say...it hit me pretty hard.
Suddenly the installer finished not once, not twice but five times in a row installing FreeNAS without any interruptions. I rebooted, gparted the disk and installed again and every time it worked.

I mean wtf?

So I double checked, enabled all cores again and here we are again fellow checksum error.
I disabled all but one core - installer worked like a charm, I rebooted and confirmed with a second installation run.
I went on with two cores - installer worked just perfect, again confirmed it a second time.
I enabled all but one core - it just worked, multiple times confirmed
I enabled all cores - boom, checksum error

Seriously, wtf?

Ok, now I enabled all the features again but just went single core. It worked
all features + duo core. check
all features + three cores. check
all features + all cores. NOPEEE! checksum errors again

...I'm tripping...

I went on with trying different FreeNAS isos. When I started this thread I had tested multiple iso of different releases from the download archive. I had trouble installing 11.3, 11.2 U7, U6, U2, only 9.10 would work - god knows why. Now all of the releases worked like a charme. No checksum errors, no aborts, nothing, as long as I deactivate one core in the BIOS.
I even tried updating from 11.3 to 11.3 U1 via the webGUI and it worked out well.

I'm pretty baffled, this was all not anticipated when I started to build my first FreeNAS system.
Thanks to all who have helped or tried to help!

all the best!
#mm

 

[mistermanko]

Polite bump :)

 

[G8One2]

I'm no expert, but if i remember correctly, FreeNas no longer recommends installing to USB flash drives. Rather, it should be installed via SSD or SataDOM. There was some threads about it in the past, I think it had to do with compatibility with FreeNas and that USB don't truly give full access to the disk FreeNas requires. Hopefully someone smarter than I can articulate this better.

 

[mistermanko]

I'm no expert, but if i remember correctly, FreeNas no longer recommends installing to USB flash drives. Rather, it should be installed via SSD or SataDOM. There was some threads about it in the past, I think it had to do with compatibility with FreeNas and that USB don't truly give full access to the disk FreeNas requires. Hopefully someone smarter than I can articulate this better.

Yes, the issue with USB flash drives - as I've read - is they get destroyed rather quick because the newer FreeNAS iterations rely on heavy I/O activity on the boot drive. That's why USB sticks are not recommended to use.
But that does not explain why I'm not even capable to install on to a USB drives. Or a hard drive. Or a SSD.

 

[G8One2]

Yes, that's a weird issue. Maybe a BIOS setting? Are your controllers set to AHCI?

 

[mistermanko]

Yes AHCI is enabled and after restoring the BIOS settings to defaults, I doubt it's the bios.
Again, writing the 9.10. iso to the same USB sticks and sticking it in the same USB 2.0 slots just works.

 

[aaron.stjohn]

Hello, @mistermanko. Thanks for reaching out to us on the forums and reporting this. I have filed a bug ticket in the FreeNAS Jira project. The bug ticket can be found here. Unfortunately, I do not have the necessary expertise to try and solve this issue, but I have put it in the right hands.

Thanks again.

 

[mistermanko]

Thank you very much for doing so, @aaron.stjohn.
I'll will further investigate this issue this afternoon and perhaps add more info to the bug ticket.

 

[Kris Heslop]

I am having the same problem, posted the details I could to the bug report, noting the error message and progress obtained prior to error message.

I would second the comments above that this error is more likely from a download/file checksum error VS a disc corruption error.

 

[mistermanko]

@Kris Heslop did you attempt to do a fresh install with the iso from the website?

 

[mistermanko]

I want to give a little update to whoever might be intererested:

I had a little time at the weekend so I decided to go on with further testing,
One thing is for sure, my mainboard has issues with installing any FreeNAS.iso later than version 9.10.

What I can rule out:
it's not a failed download,
it's not Legacy vs UEFI, both doesn't help installing, nonetheless booting the installer works fine with both,
it's not my USB thumb drives, because they install FN just fine via another mainboard,
it's not the SSDs, they also install flawlessly via the other board,
it's not the BIOS, I re-installed the latest BIOS and the lateste BMC FW, resetted everything to defaults,

BUT: I managed to get some successful results.

It was possible to install from USB onto a SSD via SATA on-board, when nothing else was installed/attached, not even the HBA.
It's not consistent, it wouldn't work everytime but it definitely did never work when my HBA was installed.

With that, I switched through every SATA-port and every USB slot my board provides and still did not get consistent installs.
So, next thing what came to my mind was, I never really switched off the PSU after restarting to get another install-attempt runnning.
Maybe the RAM or some caches still had corrupted information from the last install and interferred with the next install, I thought.

So everytime between install-attempts I completly switched off the PSU for a minute and tried again. I'm not sure if it was helping because I still did not get an install everytime. Sometimes it worked, like 1 out of 10, the other 9 times it just throws the failed checksum error at me.

It's beginning to dawn on me, that my board has some kind of weird glitch.

 

[mistermanko]

Update to 11.3U1 via GUI not possible. Manual update install stops with the following error message:

Code:

[EFAULT] %Packages/base-os-11.3-U1-b73ae33bad2e89086140af9fead6274c.tgz has invalid checksum

I had multiple atempts, via Brave browser from MacOSX High Sierra
via Firefox on a Linux Live USB.

WTF is wrong with my system?

The manual install eventually did work out. Now on 11.3 U1.

 

[rcd]

I'm no expert, but if i remember correctly, FreeNas no longer recommends installing to USB flash drives. Rather, it should be installed via SSD or SataDOM. There was some threads about it in the past, I think it had to do with compatibility with FreeNas and that USB don't truly give full access to the disk FreeNas requires. Hopefully someone smarter than I can articulate this better.

Sorry but where have you got that from? The 11.3 installation guide quite clearly suggest installing FreeNAS to USB sticks. Both Points 2 and 2.3.3 suggest it, I don't see anything about it being discouraged?

Not trying to start an argument here, just want to make sure I'm doing the right thing, installing it on a set of mirrored USB sticks.

 

[rcd]

Apart from that, I'm here for the same reason as the OP:

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 349, in run
    await self.future
  File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 388, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *([self] + args))
  File "/usr/local/lib/python3.7/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
    return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.7/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 964, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/update.py", line 572, in download
    get_handler=handler.get_handler,
  File "/usr/local/lib/freenasOS/Update.py", line 1100, in DownloadUpdate
    ignore_space=ignore_space
  File "/usr/local/lib/freenasOS/Configuration.py", line 1273, in FindPackageFile
    raise pkg_exception
freenasOS.Exceptions.ChecksumFailException: %Packages/base-os-11.3-U1-b73ae33bad2e89086140af9fead6274c.tgz has invalid checksum

 

[mistermanko]

Sorry but where have you got that from? The 11.3 installation guide quite clearly suggest installing FreeNAS to USB sticks. Both Points 2 and 2.3.3 suggest it, I don't see anything about it being discouraged?

Not trying to start an argument here, just want to make sure I'm doing the right thing, installing it on a set of mirrored USB sticks.

Hi @rcd, there are dozens of threads here where people complain about dying USB-sticks. Since v11 FreeNAS performs a large amount of writing operations on it's boot drive which ends most USB drives pretty fast. If you are determined on booting from USB, mirror two of them at least.

Apart from that, I'm here for the same reason as the OP:

Can you run

Code:

freenas-verify

in the shell please?
Expected return should be:

Code:

All Files, Directories and Symlinks in the system were verified successfully

 

[twitchingtongues]

I just ran into this same issue today while trying to reinstall after one of the USB sticks in my mirror died. Decided to fresh install on 2 new USB sticks.

Same motherboard as you but with an i3 instead of a Xeon. Disabling CPU cores made no difference. Same error.
I also detached the USB 3 header and put the install USB into a USB 2 slot.
Kind of at a loss now. This machine has been running FreeNAS for about 5 years until now.

 

[mistermanko]

Hey @twitchingtongues let's run through some troubleshooting steps:

• have you tried to install in BIOS/legacy mode instead of UEFI? (I mean when the installer asks if you want to install in UEFI or legacy mode)
• does the installer always stop at "invalid checksum error" or is it sometimes just freezing while showing "active set on da2"?
• after an unsuccessful attempt do you gpart delete + destroy the boot medium before you try again?

 

[twitchingtongues]

Hey @twitchingtongues let's run through some troubleshooting steps:

• have you tried to install in BIOS/legacy mode instead of UEFI? (I mean when the installer asks if you want to install in UEFI or legacy mode)
• does the installer always stop at "invalid checksum error" or is it sometimes just freezing while showing "active set on da2"?
• after an unsuccessful attempt do you gpart delete + destroy the boot medium before you try again?

Hey @mistermanko, I haven’t tried BIOS but I will.

Always stops at invalid checksum

After an unsuccessful attempt I’ve been putting both USB drives in my MacBook and doing an Erase with the Drive Utility.

 

[mistermanko]

@twitchingtongues any news?

 

[twitchingtongues]

@twitchingtongues any news?

@mistermanko I put a hold on reinstalling for a moment and decided to put the old boot stick back in and add a new one to the existing boot-pool, replacing the degraded one.
After a resilver it reported that new USB stick as faulted as well.

My plan at the moment is to add an SSD to the existing boot pool and let it resilver.

When I do that I'm going to try a fresh install one more time and select BIOS instead of UEFI to see if that makes a difference.