some input on this UI crash

[micahtangelo]

Hey all,

Had a cheapo USB boot drive in a production system that was throwing critical warnings, so replaced it with a Sandisk Cruzer - initially experienced a few of the "CCB request completed with an error" but after a cold reboot, those errors didn't come back. Restored backed-up config, everything seemed fine, and then about 8 hours later, after a busy day for the system, a coworker was trying to add a user and things seemed very slow; the UI had at some point stopped responding normally. Could open it and navigate to some areas but not to others, couldn't do a lot of basic stuff. File services kept on trucking, so users weren't experiencing problems, but freenas clearly in trouble. This evening I rebooted it, came right back up and seemed nominal; passed a boot verify and a boot scrub no issues.

So I'm hoping for some input from the community on what might have caused this problem, here is the /var/log/messages file from the time of the crash: http://dpaste.com/13SE69K

Thanks!

 

[southwow]

Looks like one of the python files for the GUI is missing. You probably need to reinstall your USB boot drive.

 

[micahtangelo]

do i misunderstand that the 'verify' fucntion does a checksum on the entire boot filesystem, and would uncover any missing file? the usb passed a verify and a scrub, and the UI seems to be doing fine now.

i've read scattered reports of possible issues with the cruzer fit USB and supermicro x9 boards. we've seen this error on two supermicro x9 machines with cruzer fits. perhaps there's a special quirk that can be manually aplpied to these drives? anybody else had this issue, including the CCB errors, and know a workaround?

 

[southwow]

I have a 16gb cruzer fit and have had 0 issues. Very strange. Following this one!

edit: this guy ---
https://www.newegg.com/Product/Product.aspx?Item=N82E16820173033

 

[micahtangelo]

here are some examples of the scattered reports.

https://forums.freenas.org/index.php?threads/sandisk-cruzer-fit-boot-problems.40329/

https://forums.servethehome.com/index.php?threads/freenas-install-fail.10607/

https://redmine.ixsystems.com/issues/7435 (may be related, may not - errors are different but behavior oddly the same)

https://forums.freenas.org/index.ph...a-few-clarifications.27870/page-2#post-184109

 

[micahtangelo]

i really like the cruzer fit drives, but if they have issues with some boards/configs/firmwares whatever, there maybe should be a note on the .... </me goes to find a link to the recommended usb drive page, and finds it's not from freenas but from sth! aah! https://www.servethehome.com/buyers...as-nas-servers/top-picks-freenas-boot-drives/ >

and in fact the only guidance officially from freenas is "high-quality USB sticks" - perhaps a note about, if we gather enough negative information, the sandisk cruzers possibly creating issues in certain combinations may be useful. maybe not. http://www.freenas.org/blog/a-compl...ware-design-part-iv-network-notes-conclusion/

 

[micahtangelo]

note, i'm wrong about my drives - they're Sandisk Ultra Fit , SDCZ43-032G-Q46 https://www.sandisk.com/home/usb-flash/ultra-fit-usb

 

[southwow]

note, i'm wrong about my drives - they're Sandisk Ultra Fit , SDCZ43-032G-Q46 https://www.sandisk.com/home/usb-flash/ultra-fit-usb

The one I posted above is an ultra. I'm using them in x9 and x10 systems. Very strange.

 

[Redcoat]

Many users here, me included, have experienced short life and multiple failures of USB boot drives in the last year or so with recent versions of FreeNAS. You have not told us yet what version(s) of FreeNAS your servers are running, BTW.

There are many theories on the USB stick failures - among them are: compact design of drives leading to poor cooling capability and resulting heat stress, crap execution by manufacturers, drives being incapable of accepting the frequent writes of system dataset updates without failure, etc., etc., etc. Search these forum postings and you'll find lots of reports. USB3 drives seemed to be more prone to failure than USB2 perhaps. From my own experience and observations, drive quality was not a necessarily good indicator - I have a bag full of 32GB Cruzer Fits that my backup server died on.

This problem has been frequent enough and annoying enough for the USB boot drive to be abandoned by many of us who experienced these failures.

Accordingly, many of us have installed SSD's as FreeNAS boot drives and eliminated the problem. Of course, this solution requires a free SATA port for the drive. An alternative, also where a SATA port has been available, has been to re-purpose a used SATA hard drive from an upgraded laptop.

So, I suggest that you at least prepare and install a new boot USB - and perhaps make a couple of spares while you are at it. Better yet, get and install an SSD for each server if you have available SATA ports.

 

[micahtangelo]

You have not told us yet what version(s) of FreeNAS your servers are running, BTW.

omg you're right, sorry!
This was a brand-new install of latest FreeNAS as of last week, 11.1-U1 iirc. It's on a Supermicro X9Dri board with dual E5-2640s, 256gb ram, 4x 500gb samsung 960 pro nvme drives, 4x 2tb samsung 960 pro nvme drives, and 8x 3TB WD greens. I didn't spec the drives, don't hate me. the 2tbs are all l2arc and the 500gbs are all slog. This is an initial foray into the area of high-performance FreeNAS, we've also got a Chelsio 40G card in here that I'm trying ot figure out why we're getting such horrendous performance from (totally different topic, and I have a thread on it if you search 40G).

As soon as we can (this machine is in another country) we're going to get someone in to drop a second new USB (maybe a different make/model!) for a mirror, or maybe I'll just do another fresh install and re-mirror to the current device.

In any case I don't mind not using USBs but the official hardware recommendations say a high-quality USB is fine, which is why we went with it. why not? i figured. well, maybe this is why.

we're probabaly just going to start running satadoms instead for primary systems and relegate USB boots to backups (maybe). does anybody have experience with the cheap chinese satadoms, just out of curiosity? they're like half as much as the supermicro branded ones, that's why i ask.

example: https://www.aliexpress.com/item/For...B-2GB-4GB-8GB-16GB-lower-price/502503839.html

2 year warranty, hmm

 

[Redcoat]

Satadoms are pricey here compared with SSD's. Used SSD's are plentiful and cheap and are fine as boot devices for personal use at least. Mirroring of SSD's is done but often spoken of as overkill.

Here's today's manual section on boot devices:

1.4.2. The Operating System Device
The FreeNAS® operating system is installed to at least one device that is separate from the storage disks. The device can be a SSD, USB memory stick, or DOM (Disk on Module). Installation to a hard drive is discouraged as that drive is then not available for data storage.

Note

To write the installation file to a USB stick, two USB ports are needed, each with an inserted USB device. One USB stick contains the installer, while the other USB stick is the destination for the FreeNAS® installation. Be careful to select the correct USB device for the FreeNAS® installation. FreeNAS® cannot be installed onto the same device that contains the installer. After installation, remove the installer USB stick. It might also be necessary to adjust the BIOS configuration to boot from the new FreeNAS® boot device.

When determining the type and size of the target device where FreeNAS® is to be installed, keep these points in mind:

• The absolute bare minimum size is 8 GB. That does not provide much room. The recommended minimum is 16 GB. This provides room for the operating system and several boot environments created by updates. More space provides room for more boot environments and 32 GB or more is preferred.
  
  
• SSDs (Solid State Disks) are fast and reliable, and make very good FreeNAS® operating system devices. Their one disadvantage is that they require a disk connection which might be needed for storage disks.
  
  Even a relatively large SSD (120 or 128 GB) is useful as a boot device. While it might appear that the unused space is wasted, that space is instead used internally by the SSD for wear leveling. This makes the SSD last longer and provides greater reliability.
  
  
• When planning to add your own boot environments, budget about 1 GB of storage per boot environment. Consider deleting older boot environments after making sure they are no longer needed. Boot environments can be created and deleted using System → Boot.
  
  
• Use quality, name-brand USB sticks, as ZFS will quickly reveal errors on cheap, poorly-made sticks.
  
  
• For a more reliable boot disk, use two identical devices and select them both during the installation. This will create a mirrored boot device.

 

[micahtangelo]

Yep, so it says there pretty much the same as what it says in the "Complete Guide to FreeNAS Hardware Design" conclusion: http://www.freenas.org/blog/a-compl...ware-design-part-iv-network-notes-conclusion/

Boot Devices
FreeNAS was originally designed to run as a read-only image on a small boot device. The latest versions now run read/write using ZFS. A SATA DOM or small SSD is a great boot device for the latest versions. Since ZFS is used, the boot device itself can be mirrored. As an alternative to a SATA DOM or SSD, one or more high quality USB sticks can be used. As an absolute minimum, the boot device must be 4GB, however 8GB is a more comfortable and recommended minimum. Beyond 16GB in size, the space will be mostly unused. Since the boot device can’t be used for sharing data, installing FreeNAS to a high capacity hard drive is not recommended.

i.e., high-quality/name-brand USB sticks are fine. Except that, apparently, users occasionally and mysteriously experience serious production-stopping issues with USB boot drives, including brand-new high-quality brand-name USBs, like we are here with several supermicro x9 systems.

Enough people on these forums with huge amounts of experience and wisdom have commented that they basically refuse to use USB boot media on important systems, but that wisdom is not reflected in the docs or hardware guide.

What would be the process to gather enough evidence that a note in some official place be made that USB boot drives should be treated with caution for primary systems for businesses? We're trying to build a storage practice around FreeNAS including TrueNAS units whenever we can, and would have liked to be able to build systems that don't experience unexpected issues even when following the guides and docs. Maybe I'm just too obtuse, but the hints suggesting SSDs are preferred seem very light to me.

 

[wblock]

That is a three-year-old blog post. As with all blogs, things can change after it is published. The current User Guide does recommend an SSD, along with updated sizes: http://doc.freenas.org/11/intro.html#the-operating-system-device.

 

[micahtangelo]

just a quick update, experiencing the same issues with the same USB drive in a different motherboard, an HP storeeasy 1430 that is scavenged. so it's not limited to just the supermicro boards, seems to (possibly) be a problem with the sandisks? or just USBs in general. or perhaps i've just been exceedingly unlucky.

that same link does say, "Use quality, name-brand USB sticks, as ZFS will quickly reveal errors on cheap, poorly-made sticks."

and doesn't give any warnings that even good quality USBs fail as often as they seem to. perhaps an updated warning is reasonable?

EDIT: added info about new system and cleaned up quote from doco.