SOLVED Very slow boot and shutdown (and reboot) after upgrade to 11.2-U7 from 11.1

[mgd]

EDIT:
See the conclusion at the bottom of this post with instructions.

Quick summary: 11.2 requires more from its boot devices compared to 11.1. If you experience very long boot times after upgrading to 11.2, then replace your boot devices with SSD drives.

---

ORIGINAL QUESTION

I have a FreeNAS system based on

• ASRock Rack C2550D4I
• 16GB DDR3 ECC RAM
• 2 x 4TB WD RED drives (mirrored data pool)
• 2 x 16 GB Kingston DataTraveler Micro (mirrored boot devices)

I recently had the motherboard replaced under warranty (after taking part in the Intel Avaton C2000 mass die-off) and yesterday I turned on the system and immediately upgraded to 11.2-U7.

Boot
Now, the system boots and works but is really slow booting and shutting down. After power on and after the motherboard is done with POST (which takes some time) the boot begins and seems to progress fine with messages scrolling down at the console.

However, when I get to the message…

Trying to mount root from zfs:freenas-boot/ROOT/11.2-U7 []...

…I am stuck here for minutes (we are talking about more than 5-10 minutes).

After that the boot continues by loading plug-ins which progresses but much slower than the initial part of the boot. Eventually, the system comes online.

Shutdown / Reboot
When shutting down the system I see loads of messages like this:

freenas collectd[...]: rrdcached plugin: Failed to connect to RRDCacheD at unix:/var/run/rrdcached.sock: Unable to connect to rrdcached: No such file or directory (status=2)

Finally, the system is syncing disks and reports All buffers synced. and then I have to wait a long time for the system to actually turn off or start to boot.

Ideas?
I have tried to check if there is something wrong with the boot pool but it seems to look fine and was scrubbed yesterday:

Code:

$ zpool status freenas-boot
  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0 days 03:45:00 with 0 errors on Sun Dec 22 07:30:01 2019
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        da1p2   ONLINE       0     0     0
        da0p2   ONLINE       0     0     0

errors: No known data errors
$

However, it looks like a scrub of a 16GB ZFS pool taking almost 4 hours is pretty slow.

Any ideas to debug and resolve this problem is highly appreciated.

---

EDIT:
I checked out this thread
* https://www.ixsystems.com/community/threads/rrdcached-plugin-failed-to-connect.76516/
as it seemed related.

After checking the checkbox “System > System Dataset > Reporting Database” I might have gotten rid of the “RRDCacheD” messages. However, the system is still taking around 20 minutes to startup. And now I have these messages:
Error: attempt to write to a readonly database

This led me to this thread which suggests 11.2 has a serious problem booting of USB devices:
* https://www.ixsystems.com/community...r-attempt-to-write-a-readonly-database.70827/

Any comments?

---

CONCLUSION

FreeNAS 11.2 needs a fast boot device – Use SSDs
Apparently, FreeNAS 11.2 requires more from its boot devices that the previous version 11.1 did. If you have a slow USB boot device (maybe it has become slow over the years due to wear) this is the time to replace it to an SSD boot device. These are so inexpensive now that here in Denmark I could buy two 120 GB Kingston SSDNow A400 drives and a SATA power splitter for 400 DKK ≈ 59,50 USD ≈ 53,50 EUR. Replacing my 16 GB mirrored USB boot pool with these two drives gives me a 120GB boot pool instead – much more than I need but it allows for wear leveling over the SSD to make it last much longer

Preparation
Before replacing your boot pool, make sure to

• Export your FreeNAS configuration from System > General and pressing the Save Config button. Consider ticking Export Password Secret Seed if you have stored passwords for 3rd parties (e.g. SMTP configuration)
• Important: Download your encryption keys if you have encrypted pools under Storage > Pools. (Read exactly how in the manual)
• [Optional]: Take a backup of the SSH host keys in /usr/local/etc/ssh/ssh_host_*

Now, you have what you need if something goes wrong and you need to do a complete reinstallation of your boot device.

Upgrading the boot pool to SSD
We are now ready to upgrade the boot pool. Follow these steps:

• Physically install the two new SSD devices in your system
• Boot the system
• Go to System > Boot Environments and click Boot Pool Status
• If your boot device is already mirror, you need to detach one of the devices from the boot pool because FreeNAS unfortunately does not support 3-way mirrors: Click the three vertical dots next to one of the devices and select Detach
• Attach the first SSD:
  • Click the three vertical dots next to the Pool and click Attach
  • Select one of the new SSDs in the dropdown
  • Remember to tick “Use all disk space” because otherwise you boot device will not grow.
  • Click Save
• Wait for the mirror to completely resilver. There is a spinning wheel in the top bar of the GUI as long as the resilver takes place. Please note, this can take very long time if your USB is very worn. For me it took 4 hours to resilver 7.5 GB of data.
• Detach the last USB drive: Click the three vertical dots next to the USB device and select Detach
• Attach the second SSD just like you attached the first. Remember to tick “Use all disk space”.
• Wait for the mirror to completely resilver. This is fast. For me it took around 2 minutes.
• Shutdown your NAS, remove the USB pen drives and restart.

Now, we are done and your pool is running on SSDs, has increased to the full capacity of the SSDs and your boot should be fast. My boot time went from > 25 minutes to around 2.5 minutes.

 

[Redcoat]

Yes - FreeNAS is a server OS. Implicit, typically, is that booting up (power-on and power-off, too) is infrequent. Why are you hung up on this issue? Do you have error messages?

 

[mgd]

I know. Normally, I never reboot my FreeNAS system unless I need to. I basically let it run and only reboot (automatically) when there is a power outage.

In this case, however, my C2550D4I motherboard died and was replaced by ASRock under extended warranty. When I got it back I thought it was a good time to upgrade to 11.2 which have now been out for some time. The upgrade took for ages including the final startup.

You are right, that when the system runs (and runs properly) it shouldn't matter (that much) how long it takes to boot.

However, 25 minutes for a boot from power off is ridiculously long. Also, this is a symptom of something being wrong and as noted above, I now have these error messages in massive numbers:
Error: attempt to write to a readonly database

Furthermore, after one reboot, my encrypted pool did not come online automatically and was still locked. Rebooting once more resolved this.

All this seems to be an issue of what is discussed in this thread which I did not find until this evening:

Fresh install 11.2 on USB drives, terrible performance

Greetings, We rented a new server to be used for backups. This server has this motherboard and we're using two Kingston Data Traveler 3.0 64GB sticks which are plugged into the USB 3.0 ports I believe. This is a setup we've been running for over a year on different Supermicro hardware without...

www.ixsystems.com

So yes I am worried that this will cause me further problems along the way because it seems that after this having been an issue for a year now and nothing has been done about it, there is probably no testing on USB boot devices from IX Systems which means future issues related to USB boot devices will not be found before release of a new version.

I guess my options are:

• Downgrade to 11.1 or
• Replace my USB boot devices with SATA SSD drives.

Comments?

 

[jon_2112]

Did you try a fresh install of the OS onto the USB drive(s)? I did fine booting 11.2-U7 off USB drives.

 

[Redcoat]

Sorry - I missed the error message report in your post.
I had 3 C2750D4I replacements with my FreeNAS Mini so I feel for you in this process. But I did not have encrypted pools so I have no idea how this may influence your experience.
I definitely support a change to SSD from USB boot.

 

[mgd]

Did you try a fresh install of the OS onto the USB drive(s)? I did fine booting 11.2-U7 off USB drives.

I will try that (both UEFI and BIOS) on a blank USB stick and I will also try to boot into my old 11.1 environment on my current mirrored USB sticks. If 11.1 boots fine on the same sticks with the FreeBSD/FreeNAS bootloader (that replaced GRUB in 11.2) I think it points to the FreeBSD kernel's USB driver.

I had 3 C2750D4I replacements with my FreeNAS Mini so I feel for you in this process.

It seems like I have been lucky compared to you. :)

But I did not have encrypted pools so I have no idea how this may influence your experience.

I don't think it has any influence on this. The boot is fast right up to the point where it mounts the USB ZFS root device. That takes 20 minutes and after that the boot is still slow. Then it imports the storage pools which only takes seconds.

I definitely support a change to SSD from USB boot.

Yes, I guess this is a good idea in any case. The reason booting from USB was a good idea is that you don't need two extra SATA ports that could have been used for storage. However, the C2550D4I has 12 SATA ports, so I am fine for now.

 

[toadman]

On a glance, I would have guessed one of the USB drives was bad. Try replacing one or both and see if that solves your problem. Or switch to the SSD boot if desired.

 

[mgd]

As I wrote above, I will try an install on another USB stick (single stick, non-mirrored, keeping my existing boot media USB sticks untouched) to see how it works out. This other thread indicates that the USB sticks are not the problem:

Fresh install 11.2 on USB drives, terrible performance

Greetings, We rented a new server to be used for backups. This server has this motherboard and we're using two Kingston Data Traveler 3.0 64GB sticks which are plugged into the USB 3.0 ports I believe. This is a setup we've been running for over a year on different Supermicro hardware without...

www.ixsystems.com

But of course you never know. :)

 

[toadman]

Ah. Thx for the pointer to that issue. I hadn't see that before. I'm running FreeNAS in a VM, so I haven't dealt with a bare metal boot in a long while.

 

[mgd]

Quick update
It seems FreeNAS 11.2 simply requires a much faster boot device than 11.1 did. I will switch to SSDs and see if it helps.

I did a thorough investigation with a fresh install of 11.2 with BIOS and UEFI boot devices and timed it against my existing environment. The measurements and conclusions are posted in detail here:

Fresh install 11.2 on USB drives, terrible performance

I'm not sure how that would help. That said, this is rented hardware we can't modify. These things can be plugged into external USB jacks, so they are like a big wide USB stick. Haven't tried one in a FreeNAS system, but they do work, often much faster than USB sticks...

www.ixsystems.com

 

[mgd]

After installing two new SSD drives as mirrored boot device a complete boot measured from after UEFI POST and until the system is completely ready now takes less than 2.5 minute.

When installing the SSDs I detached one USB boot drive and attached one SSD at a time to the boot pool. Resilvering from the remaining USB drive to the first SSD took almost 4 hours and resilvering from the 1st SSD to the 2nd SSD took 2 minutes and 18 seconds, so I guess the USB drives have been close to dying.

Thanks @Redcoat, @jon_2112 and @toadman for your help!

 

[Redcoat]

Thank you @mgd for taking the time to document and describe this issue so diligently. This should help newcomers make good decisions on boot devices.