[Unpopular Opinion] Are USB sticks really *that* bad for the boot-pool, even today?

[winnielinnie]

The title of this thread doesn't do the question justice, but in summary, let's say you have the following setup with your TrueNAS server:

• boot-pool (i.e, "OS") is installed to a USB stick
• No swap is used on the USB stick
• System Dataset is moved to your data-pool
• Syslogs are written to your System Dataset, which means they write to your data-pool
• Swap is used on your data drives (default 2GB)
• You regularly backup your config file and any encryption keyfiles

So how is this risky or frowned upon? Barely any writes at all are done to your USB stick (no syslogs, no System Dataset, no swap), and if your USB ever fails, you only lose the OS, which is easy to bounce back from: Reinstall TrueNAS. Upload your config. Done. (You can even mirror your boot-pool with two USBs, and it still costs almost nothing.)

This frees up ports on your motherboard and/or controller, and means you needn't purchase an additional drive(s) just to use as a boot-pool for something that barely ever gets written to. Not to mention, less space is taken up inside the chassis in regards to drive bays and cages. Not to mention that even if you have m.2 slots for NVMe, they can be used for more practical benefits, such as SLOG or L2ARC or a dedicate "fast scratch" or jail pool, rather than wasted on a boot-pool. That's a big deal for home users.

I switched from using mirrored-USBs as my boot-pool to now using SSDs, but I realized it feels like a pointless waste since my System Dataset and syslogs are not even on my boot-pool!

It's like owning an expensive car, which you never drive, just so you can use the interior space to store some luggage. Ouch!

 

[Constantin]

I simply go by how many folk here have reported a corrupted USB stick vs. how many report that their SSD boot pool went kaplooiee. It's not even close. All the steps you took to limit writes to the stick are a good start, but the underlying issue (the cheapest, likely recovered / desoldered Flash on the planet being used in these sticks) is not. I simply do not consider the payback of a cheap stick worth the additional time and aggravation that it likely will entail.

... and I say that as a user of SATADOMs, which arguably are very similar to USB sticks, except that they use higher-quality flash and other new, not recycled components. I don't think I will ever have to worry about a SATADOM dying on me, yet I still have a spare.

 

[winnielinnie]

the cheapest, likely recovered / desoldered Flash on the planet being used in these sticks

and I say that as a user of SATADOMs, which arguably are very similar to USB sticks, except that they use higher-quality flash and other new, not recycled components.

Isn't that the entire concept of higher-end USB sticks, such as this one? (Though 128GB is a bit overkill, the idea behind it remains.)

They plug right into an external USB port, and you have an extra free SATA port on your motherboard and/or controller. (No need to purchase an NVMe drive with a USB-adapter enclosure, as I have seen some people do.)

 

[winnielinnie]

Here is the I/O activity on my boot's SSD drive since September 2021 up until today. (Each slice of the grid is a month.)

Those blue read "peaks" are only 120KiB. The purple write "peaks" are almost nothing. Notice the timeline and how far apart these insignificant I/O operations are? (They're so small, you can't really call them I/O operations.) A student using a USB stick for their schoolwork and documents, and other casual uses, puts more stress on a USB stick than does TrueNAS using it as a boot-pool (if you don't write swap, syslogs, or System Dataset to it, which is easy to configure since it can be offset to a data pool.)

The real question is, could a USB stick, especially with high-quality components, handle this "load"? (Especially if two sticks are mirrored, you're still "up and running" if one does happen to fail.)

 

[flashdrive]

Here is what I am currently doing:

3 USB2 thumb drives as boot pool mirrored

swap 1 of the 3 drives periodically - so only 2 out of 3 are ever connected

Upon replugging auto scrub will update the USB drive content.

My mainboard only has got 6 SATA connectors - so not loosing the connectors to the boot pool was essential to me.

I did not want to go the HBA route. It's Home / SoHo usage only.

Works fine for me.

 

[jgreco]

Isn't that the entire concept of higher-end USB sticks,

Yes. No one's saying that it is impossible to do.

Unfortunately, the problem is that the vast majority of PC hardware is NOT "high end", and the stuff that IS "high end" (and therefore profitable) is often the target of knockoff fakes.

The fundamental issue is wear leveling, and if you have a high quality USB stick that has a controller in it that does some wear leveling, I would expect that you could be successful.

 

[emk2203]

A couple of things here where I think I can contribute with my own experience.

• Everyone focuses on the writes, but there's more to the fragility of a USB stick. In my experience, if you have an elevated temperature, the USB stick will die over time. My 2 microservers have an internal USB port, and I had to replace the sticks already 2 times each. As already said: The material in these sticks is crap. The level to which I trust them is that of a floppy disk of the past. On my laptop, one USB port gets even hotter, and I have dubbed it the "stick killer". Just got a replacement from SanDisk for the next dead stick...
• The Corsair one is 99% marketing and 1% crap. Sorry to be so blunt, but also based on personal experience. Not worth your money. The write times on them (and on other sticks as well) go up over time like you wouldn't believe. 'Premium' is just their ad budget.
• A backup of the config together with a fresh install is enough to bring the server up, so I don't see the value of a mirrored boot pool except for HA production where downtime costs serious money. For a SOHO scenario where you are close to the server when you notice it failed, it is good enough.
• The free ports are an issue, since for most of us, they are the bottleneck. What I did and will continue in the future: Get a M.2 to USB adapter and put a M.2 SSD in. For my SOHO case, a cheap used-but-reliable one is plenty good for the purpose. The ones I have sped up boot and upgrade times by a factor of 5 or more. Not exaggerating. Upgrades went from 20 - 30 minutes on the N40L microserver to 5 minutes. Also before: every 12 - 15 months a new USB stick. Always when I didn't have time and energy for that. Now: running like the D*rac*ll bunny for more than three years. I am not loosing precious SATA ports. I am in full control over what I put into these adapters, quality-wise.

 

[emk2203]

The fundamental issue is wear leveling, and if you have a high quality USB stick that has a controller in it that does some wear leveling, I would expect that you could be successful.

It's not just the wear leveling with the sticks. I used to have a couple of OS images (rarely any read or write) on a thumbnail stick which just fits into the USB port. Over time, this port (directly over the laptops hot GPU) kills all the sticks in there without fail. Just got the next one via manufacturer warranty. I suspect with the inferior quality, any heat degrades the flash memory.

 

[winnielinnie]

The Corsair one is 99% marketing and 1% crap. Sorry to be so blunt, but also based on personal experience. Not worth your money. The write times on them (and on other sticks as well) go up over time like you wouldn't believe. 'Premium' is just their ad budget.

Their specs show it uses 3D NAND TLC, which is basically an SSD drive in a small form-factor, that happens to use a USB type-A connector. I don't regard it as simply a "better" USB stick. The only "USB" quality about it is the connector. I also don't see it overheating if plugged into an external USB port (and with little to zero activity anyways.) Server or ATX chassis are less prone to really hot USB ports as opposed to laptops, due to the design and layout.

Wouldn't it be essentially the same as what you described later: to purchase a "good enough" m.2 SSD, and pop it into a USB adapter? This seems to be what the Corsair is, except it's all a single, convenient piece of hardware.

Did you ever use one of those Corsairs specifically, or did you have bad experience with other "gimmicky" USB sticks that tried to pass themselves off as "high quality"?

 

[emk2203]

Not this one specifically, but I had a durable Corsair stick which got almost to USB1 levels in speed with very light usage and my one Corsair SSD also malfunctioned. The only one ever that broke for me. So I am not too keen for more Corsair experiences...

The stick I had was a Corsair Survivor. Predecessor of this one:

Looks impressive, but the write times went from bad to abysmal. Could withstand probably anything -- except reads and writes.

 

[Constantin]

The real question is, could a USB stick, especially with high-quality components, handle this "load"? (Especially if two sticks are mirrored, you're still "up and running" if one does happen to fail.)

To me, a USB stick vs. SATADOM could be compared to a external hard drive vs. a bare internal. The external drive has no forced convection, and is in a tight plastic enclosure with minimal heat transfer. The internal SATADOM is bare and hence can enjoy the benefits of forced convection. The chip temperatures, all things being equal, will not be even remotely close.

SATADOMs are popular in pro servers for boot pools because they use SATA->Flash controllers that actually work reliably. The USB->Flash controller found on USB sticks will be the least expensive stuff from the least expensive supplier. No consumer has the expectation that USB flash is reliable enough to be used continuously in a server application, and OEMs build accordingly.

Flash that can be written to a lot (so-called high-endurance stuff, though usually without specs that you can rely on) is priced significantly higher than "regular" flash. Yet, I also would not rely on it in a boot pool scenario because I value my time and data pool too much. The Mini and Mini XL were fitted with a single 16GB SATADOM even if the C2750D4I could have theoretically allowed for the use of a internal USB boot stick via the internal USB pin header (item 22 on p. 8 of the user manual). Whether a USB stick could have fit in there is a different question. But iXsystems likely spent a lot more money on the 16GB SATADOM than they would have on a simple 16GB USB stick.

 

[pschatz100]

I am always amused when someone creates a post like this. Sure, using thumb drives will work - until they don't. I used to replace a flash drive at least once per year until I switched to an SSD. I suspect the problems arise with the controlers as there is no cooling in a flash drive and they aren't designed to run 24/7. Lots of people on the forum are using usb to sata connectors with small SSD's to save a sata port on the motherboard - and the cost is lower than the referenced Corsair thumb drive.

Actually, I think the biggest problem happens - not when thumb drives fail - but when they sorta fail. When thumb drives misbehave, they can create problems that are difficult to diagnose.