Clean Installation Fail on TrueNAS Mini X during "Preparing NVDIMM Configuration": chroot <tmpdir> truenas-nvdimm.py returned non-zero exit status 1

[larry-fish]

Hello,

I have a TrueNAS Mini X purchased around 2020 and I have been running TrueNAS CORE on it ever since. I used it mostly for file and media sharing and it was quite under utilized. I thought I'd give SCALE a go so I can utilize KVM and linux containers and not have an idle server.

I was running the latest TrueNAS CORE 13.0-U6.1 and it was stable prior to attempting a clean install of TrueNAS SCALE 23.10.0.1 via Device Media.

Steps to replicate

• I attempted a clean install and upgrade using the Installing From the Device Media instructions found on TrueNAS's Documentation Hub.
• I followed the TrueNAS Installer Console Setup Instructions
  • All of the steps were displayed as shown in the documentation except Step 5, I did not get to "Select Boot via UEFI at the TrueNAS Boot Mode prompt, then select OK and press Enter to begin the installation."
  • After choosing the admin password and proceeding, installation started immediately with:
  • 
  • After the extraction and performing post-install tasks, the installation failed during the "Preparing NVDIMM configuration" step:
  • 

There was very little user involvement as I walked through the installer as instructed. Is there anything I can do on my end to fix this issue. Cheers!

Hardware:

From iX System's TrueNAS mini series datasheet for the Mini X:

• Motherboard make and model
  • ??
• CPU make and model
  • Intel Quad-core C3558
• RAM quantity
  • 16 GB DDR4 with ECC (2 x 1 8GB)
• Hard drives, quantity, model numbers, and RAID configuration, including boot drives
  • 5 x 1 WD Red WD40EFRX
  • 2 x 1 Samsung 870 QVO
  • 1 x 1 (16 Gb drive included with Mini X)
• Hard disk controllers
  • ??
• Network cards
  • Standard: 4 x 1 Gigabit Ethernet LAN ports (10/100/1000)

 

[HoneyBadger]

Hey @larry-fish

Sorry to hear about the trouble you're having. Can you double-check the hash of your downloaded SCALE installation media?

It should be F56FBC3793240D22A97637C09A78B6812DB665BDF848A06545CBE9B2B1AB3D2F

You can check this with the Powershell Get-FileHash command under Windows, or sha256sum on Linux.

 

[larry-fish]

Hiya!

Prior to installation I verified the file hash and PGP signature. For my sanity I ran through it again and I confirm the hash matches:

Code:

[truenas_scale]$ sha256sum TrueNAS-SCALE-23.10.0.1.iso
f56fbc3793240d22a97637c09a78b6812db665bdf848a06545cbe9b2b1ab3d2f  TrueNAS-SCALE-23.10.0.1.iso

I also created the bootable usb using dd and then another time using balenaEtcher, as balenaEtcher does a verification and checks for any difference between what was written to the target device, and what was read back afterwards.

 

[HoneyBadger]

@larry-fish Are you able to try downloading an earlier version of SCALE 22.12 (or even CORE, which you could upgrade to SCALE afterwards?) and installing that clean, so that we can collect a debug bundle and/or dmidecode information?

 

[larry-fish]

@HoneyBadger I tried your suggestions and the results of them are below.

I have also attached the dmidecode outputs on the CORE and SCALE systems, along with their diff. Looks like the formatting of the SMBIOS is a bit different, otherwise everything else is the same.

As for debug logs, you'll have to tell me exactly what you need, from what system, during which state. I will keep my system offline for the week so I can get you anything your team would need before I put me data back on there.

[FAILED] Performing a clean install using an earlier version of TrueNAS-SCALE-22.12.4.2​

1. Failed at a similar error (screenshot attached + bonus cat). I looked at the truenas-nvdimm.py python script in your middleware repo and based on the python stack trace it appears to be looking for an attribute "major" for the "LegacyVersion" object to compare it to "GEN3_MIN_VERS", but the object doesn't have the attribute so it fails.
2. You guys don't have issues enabled in the github repo so I could not open an issue but here is a possible fix for the failure. You'd just have to populate how to handle the case when the version doesn't conform to the expected format:

Code:

def main():
    prod, vers = parse_dmi()

    gen1_2 = gen3 = False
    if not all((prod, vers)) or not prod.startswith('TRUENAS-M'):
        return
    try:
        parsed_version_major = version.parse(vers).major
    except AttributeError:
        # Handle the case where the version string does not have a 'major' attribute
        parsed_version_major = None

    if parsed_version_major == GEN3_MIN_VERS:
        gen3 = True
    elif parsed_version_major is None:
        # Handle versions that don't conform to the expected format
        # I'm not familiar with the install process to decide how to treat such cases, here I assume they are gen1_2
        gen1_2 = True
    else:
        gen1_2 = True

    # Rest of the code...

[SUCCEEDED] Performing a clean install of TrueNAS-13.0-U6.1​

• I was able to successfully install CORE and then upgrade to the TrueNAS-SCALE Bluefin Train from the CORE UI.

 

[HoneyBadger]

As for debug logs, you'll have to tell me exactly what you need, from what system, during which state. I will keep my system offline for the week so I can get you anything your team would need before I put me data back on there.

Hey @larry-fish

I think I have what I need from the dmidecode snippets you shared already. I'll shoot you a DM with some further info, but I've removed the raw files themselves from your post.

 

[HoneyBadger]

Looks like there's a bit of a logic fault in the installer script. I'll see about getting in a ticket to fix this, aiming for a correction in 23.10.2 - will update this post with a URL once the ticket is filed.

 

[larry-fish]

So after some stable usage on TrueNAS-22.12.4.2 Bluefin Train, I decided to upgrade to TrueNAS-23.10.1 Cobia Train. It looks like during the upgrade, the same faulty logic is hit in the installer script (see below). So if I want to upgrade Trains on your Mini-X product I have to wait until this is addressed OR I have to reinstall CORE again, and from CORE migrate to SCALE Cobia Train. However, the Migrating TrueNAS CORE to SCALE documentation has the following line the makes me doubt I can cleanly upgrade CORE -> SCALE Cobia.

"To migrate from CORE to SCALE Cobia, first upgrade to and migrate from the latest CORE release to the latest release of SCALE 22.12 (Bluefin). While running SCALE Bluefin, take the necessary steps to transition from these deprecated services to the applications that replace them, then upgrade to SCALE 23.10 (Cobia). The replacement applications are available in SCALE 22.12 and 23.10."

Does this only affect those with deprecated services or is this for all CORE -> SCALE Cobia upgrade paths (if on a clean CORE install)? This question was also asked by another user in this thread and at the time of me writing this, their question was not addressed. I suppose I'll try CORE -> SCALE Cobia myself and see if it is stable but it would be comforting to have confirmation from iX that this is indeed safe.

Lastly, is there a place I can track this bug? I did not see it filed in your external facing issue trackers.


Error

Code:

[EFAULT] Command ['chroot', '/tmp/tmpa004yqun', '/usr/local/bin/truenas-nvdimm.py'] failed with exit code 1: Traceback (most recent call last): File "/usr/local/bin/truenas-nvdimm.py", line 71, in main() File "/usr/local/bin/truenas-nvdimm.py", line 38, in main elif version.parse(vers).major == GEN3_MIN_VERS: ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/packaging/version.py", line 52, in parse return Version(version) ^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/packaging/version.py", line 197, in __init__ raise InvalidVersion(f"Invalid version: '{version}'") packaging.version.InvalidVersion: Invalid version: 'To Be Filled By O.E.M.' 

Traceback

Code:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 427, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 465, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1379, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/update.py", line 285, in update
    await self.middleware.call('update.install', job, os.path.join(location, 'update.sqsh'), options)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1368, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1328, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1231, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/install_linux.py", line 32, in install
    self.middleware.call_sync("update.install_scale", mounted, progress_callback, options)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1398, in call_sync
    return methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/install.py", line 52, in install_scale
    self._execute_truenas_install(mounted, command, progress_callback)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/install.py", line 83, in _execute_truenas_install
    raise CallError(result or f"Abnormal installer process termination with code {p.returncode}")
middlewared.service_exception.CallError: [EFAULT] Command ['chroot', '/tmp/tmpa004yqun', '/usr/local/bin/truenas-nvdimm.py'] failed with exit code 1: Traceback (most recent call last):
  File "/usr/local/bin/truenas-nvdimm.py", line 71, in <module>
    main()
  File "/usr/local/bin/truenas-nvdimm.py", line 38, in main
    elif version.parse(vers).major == GEN3_MIN_VERS:
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/packaging/version.py", line 52, in parse
    return Version(version)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/packaging/version.py", line 197, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: 'To Be Filled By O.E.M.'

 

 

[HoneyBadger]

@larry-fish Seems like the same problem.

Since the system is running at this point, can you gather a standard debug from System -> Advanced -> Save Debug, and then create a case from the "Report a Bug" link:

TrueNAS - Issues - iXsystems TrueNAS Jira

ixsystems.atlassian.net

Once you do that, you'll get a prompt from the bug tracker with a way to privately upload that debug file and associate it with the ticket. That way the iX engineering team can get a look at it and see the exact logs, without sharing anything potentially identifying publicly.

Thanks!