Fresh install 11.2 on USB drives, terrible performance

[Meyers]

Greetings,

We rented a new server to be used for backups. This server has this motherboard and we're using two Kingston Data Traveler 3.0 64GB sticks which are plugged into the USB 3.0 ports I believe. This is a setup we've been running for over a year on different Supermicro hardware without issue.

I installed a fresh copy of 11.2 on this system using both USB drives. Initial boot takes 30 minutes or more. Any configuration attempt at all locks the SQLite databases, causes UI timeouts, etc. Performance is so terrible as to make this setup completely unusable. We've had them replace our USB drives and have done several fresh installs and we get the same result every time. I've tried installing on just one USB drive and get the same results as well.

I booted SystemRescueCD and formatted these drives as XFS and ran a few basic tests i.e. dd if=/dev/zero if=test.bin bs=1024000 count=16000 (this system only has 16GB of ram). Performance was what I'd expect from a USB stick. No hangups, no 3s+ response times. It seems like for whatever reason this is a FreeNAS issue.

There are no log entries that I can see such as hardware timeouts. No other log entries other than the various middleware processes complaining about locked DB and timeouts.

Does anyone have any idea what I could do to troubleshoot this further? I'm tempted to say the heck with it and just install the OS on the data drives, which work just fine as far as I know. But this isn't a rush so I'd really like to figure out what's going on if possible.

Thanks.

 

[Meyers]

OK - I just installed FreeNAS 11.1 and everything is working as expected. Maybe tomorrow I will try upgrading to 11.2 and see what happens.

 

[Chris Moore]

This is not the first report of slow behavior with the new system software. Please give more detail about the hardware specs of your system.
Ultimately this may need a bug report.

 

[Meyers]

Motherboard / CPU: SuperMicro X10SDV-4C-TLN2F / Intel Xeon D-1521
Ram: 16GB ECC
Disk: 2 x Kingston Data Traveler 3.0 64GB (in USB 3.0 slot) for OS, 6 x HGST Ultrastar He10 8GB for data

I'll see if a bug report is already open.

 

[Meyers]

Upgraded from 11.1 and 11.2 - same issue.

 

[Constantin]

Have you tried putting the data on a SSD in an enclosure? Then try via USB?

Alternatively, hook a SATA boot SSD up and see what happens?

 

[Meyers]

These are rented servers so the build is static. USB works with 11.1 so I'll use that until this gets sorted out.

 

[Chris Moore]

Please review:
https://www.freenas.org/hardware-requirements/

Just to highlight the part I am attempting to call your attention to:

While USB boot media is still supported, it has not been the recommended option for at least a year, possibly two years. The operating system, long ago, loaded fully memory resident and the USB drive was not used after boot. That is not the case now and the drive, or drives, used for the boot pool need to be faster and higher endurance than most USB memory sticks are capable of. This change has become even more prominent with the most recent release and it is not likely to be reversed as additional features and functionality are added to the system over time. Please revise your plans accordingly.

PS. You can also use spinning disks for boot media. I have three servers that have been running happily for almost four years, each on a mirrored pair of 2.5 inch laptop drives that I bought new for that express purpose.

 

[ThreeDee]

I wonder what it is with some systems using USB sticks just fine (mine) and others (OP's) that it just cripples their systems ...

 

[Constantin]

Based on the system working fine under 11.1, it definitely seems to be software related.

Based on the lack of errors getting posted, I'd wager an ice cream sandwich that it's related to the USB driver or some other IO process changing ever so little between FreeNAS editions, causing some sort of condition that doesn't throw an error, yet also massively slows things down... like constantly triggering garbage collection, SMART tests, or other I/O processes on the USB stick that eventually conclude without error but which so thoroughly thrash the USB stick that nothing else can get done. Whatever it is, it would be deep down enough not to show during the boot process.

I wonder if posting the error / bug to the FreeNAS support site would be advantageous, especially if one of the USB sticks can be replaced with a different model and one of the originals is sent to iXsystems for replication of the issue. Come to think of it, that might be another worthwhile approach, i.e. replace one of the extant sticks with a different make and model.

 

[Meyers]

Definitely seems like an issue with 11.2. I can't do anything with the hardware since this is a rented server in a remote data center so I will just use 11.1-U6 for now. I've also filed a bug report.

By the way, we've had two production servers running in this exact setup for about 18 months now without issue (related to the USB drive anyway). I get that the USB sticks don't have near the durability that an enterprise SSD has, but with the system dataset on the data volume there doesn't seem to be much activity on the USB drives. They are mirrored as well so I think the risk is more than acceptable for us.

 

[Constantin]

Got it. When you wrote to say that you rented servers, it didn't occur to me that they might be far away in a data center. Silly Me!

All my "solutions"presumed you had physical access to try something out.

 

[181228]

I can confirm this behavior with 11.2 and USB sticks. I have two FreeNAS boxes that have been up and running for at least 5 years. They have always been run off of mirrored USB sticks without any significant problems. These boxes worked just fine until I attempted to migrate to 11.2-RELEASE. Now they behave like you described with painfully slow operation and all sorts of timeout errors. Interestingly, reinstalling 9.10.2-U6 or 11.1 on the same hardware makes the problem go away. My small sample size suggests that software rather than hardware is the root of the problem.

 

[181228]

To clarify this a bit, I've found that with 11.2 you have to pay very close attention to USB device emulation settings in BIOS. Over the course of performing multiple reinstalls I've made some interesting observations:

If my ASUS BIOS configures the USB sticks to be emulated as HDD then I suffer pronounced performance degradation. The simple process of performing a default install could take an hour instead of 10 minutes for 11.2 when USB emulation is set to HDD. System operation will slow to becoming unusable and multiple timeout errors would be issued.

If my ASUS BIOS configures the USB stick emulation as "Auto" then the slowness problem goes away. Install takes a few minutes instead of an hour and the timeout errors go away as well.

This would suggest that the OS gets information from the BIOS about the USB stick's emulated device type, and that the OS derives some sort of expectation about how fast the device should respond and what sort of time interval constitutes an appropriate timeout interval and what constitutes a timeout error. ie: a USB stick responds too slowly if it's emulating a HDD and this causes time out errors. Those same errors don't occur if the USB is configured automatically to emulate a device that is known by the OS to be slower.

Of course, you could just make the problem go away by buying an SSD, but that only ignores the actual nature of the problem.

 

[Meyers]

To clarify this a bit, I've found that with 11.2 you have to pay very close attention to USB device emulation settings in BIOS.

...

Of course, you could just make the problem go away by buying an SSD, but that only ignores the actual nature of the problem.

Hrmm...I think I have it set to auto or some equivalent. I'll double check that. I played with the USB settings a bit but not extensively.

I'd buy an SSD if I could. We have limited options with our hosting provider.

 

[pschatz100]

If your system allows, you can try setting up your system for "bios boot", sometimes called legacy boot, and then UEFI boot and see if that makes any difference. On my older X9SCM Supermicro system, USB flash drives worked fine when configured for UEFI but when I migrated my boot device to an SSD I had to install FreeNAS for bios boot. The system would not recognize the particular SSD I purchased as a UEFI boot device. I don't know why - maybe it has something to do with the controller.

 

[davbro]

I'll also chime in to note issues with 11.2 and USB system drives.

Ran mirrored USB sticks for a long time through 11.1, and upon 11.2 things became almost unworkable. This has persisted across two different systems and different USB sticks. All booting via Legacy BIOS, not UEFI.

System #1:
Xeon E3-1245 v3, 24GB ECC, Supermicro X10-SAE

System #2:
Xeon E3-1225 v5, 16GB ECC, Dell T30

A fresh install of 11.2 takes 2+ hours on both systems, if it doesn't error out beforehand. A previously installed 11.1-U5 system upgraded to 11.2 seemed OK, but took forever to boot after upgrade.

Same USB sticks will boot a Linux install at speeds you'd expect from your average USB 3.0 thumb drive.

For now I'll need to stick with 11.1-U6 until this can be ironed out, but it's definitely an issue (at least on my end, sounds like it with others as well).

Edit: per the change of recommended specs moving from USB sticks to a real drive, that's fine, but the installer itself even calls out something similar to "Installation on USB media is preferred." Stale messaging on the installer?

 

[pschatz100]

According to the 11.2-RELEASE changelog, the boot loader has changed from GRUB to the native FreeBSD boot loader. I would have thought this would make things better.

When you say a fresh install takes two hours to boot... does the system run OK after boot is completed? Have you ever tried creating a boot drive for UEFI - just to see if that makes any difference?

 

[davbro]

Funny you should mention - after making that post I tried a fresh install via UEFI, and it booted, but the same issue existed.

What I note is roughly this:

System boots more or less OK, goes through the menus asking for a root pw, where to install FreeNAS, etc. Then it goes out to format the USB sticks, the format itself goes pretty quick, and then it hangs... and hangs... and hangs. Eventually it gets to "Installing base-os..." which takes an hour or two. One or twice it's thrown an error (looked like some sort of set -x output from a shell script or similar).

Obviously at this point we're well beyond "is it booting via UEFI?" because it's handed off to the kernel in memory and gone through the installer.

I went back and installed fresh with 11.1-U6, same USB sticks, install went relatively quick. From there, I logged right into the UI, didn't set up anything else, and did an upgrade right to 11.2. Upgrade seems to have completed, but on the first reboot it hangs on trying to mount the root zpool.

If it ever gets past that, I don't know. I didn't have more than 30 minutes or so to wait for it to mount the pool.

From what I'm seeing, the problem doesn't follow the hardware or the USB sticks. It follows the 11.2 release of FreeNAS. I guess I could try a vanilla FreeBSD install just to see if it's an overall issue with FreeBSD or if it's FreeNAS-specific.

That was on the E3-1225v5 hardware.

The other machine (which I can't as easily take up and down) had basically the same result on an 11.2 upgrade from 11.1-U5. I figured something hosed, and since I had a backup of my config, I thought I'd try a fresh install. So, fresh install of 11.2, same result. I never got far enough into the system to even tell you if it would operate "normally" or not, because if it ever makes it that far it would take over an hour (and I assumed something was screwy with my hardware at the time, and didn't wait around for it).

So yeah, as far as I can tell, system operation in general via the familiar USB-stick method is compromised in 11.2.

 

[davbro]

Of course, I could have bunk USB sticks, but I've been through 4 of them. Admittedly, they were all the same size/brand, however.

They're Microcenter-branded 32GB USB 3.0 sticks. Some newish, some oldish. They all work like a champ booting up Linux and Windows install ISOs. So, I guess there's a corner case of the chipset these specific USB sticks are using not playing nice with FreeBSD... but even still, they played nice in 11.1-* and prior.