Fresh install 11.2 on USB drives, terrible performance

[pschatz100]

Of course, I could have bunk USB sticks, but I've been through 4 of them. Admittedly, they were all the same size/brand, however.

They're Microcenter-branded 32GB USB 3.0 sticks. Some newish, some oldish. They all work like a champ booting up Linux and Windows install ISOs. So, I guess there's a corner case of the chipset these specific USB sticks are using not playing nice with FreeBSD... but even still, they played nice in 11.1-* and prior.

According to the changelog for 11.2-RELEASE, the boot process was changed - so there could definitely be an issue with thumbdrives. If I were you, I would try two things: 1) try re-creating the boot drive while it is plugged into a usb2 port. Don't create mirrored boot drives - just create a single drive and see if that makes any difference, and 2) try a different brand thumbdrive (a different drive will have a different controller).

When I used to boot off of USB flash drives, I always used Sandisk thumbdrives. Often, one went bad when I tried to update FreeNAS. Sometimes, I could take the bad thumbdrive and reformat it for windows - but usually not. After I switched to booting off a small SSD, all these issues went away. I did not look into the matter any further once I started using an SSD for boot.

 

[davbro]

I'll definitely try again with a totally different USB stick and see what happens. I'll try from a USB2 port (though interestingly, FreeNAS (-BSD?) picks up the USB 3.0 thumbdrives with a 400Mbps speed, even when in a USB 3.0 port. Wondering if there's an XHCI issue. Not a big deal for the root device, though, since it shouldn't be reading/writing a lot outside of the boot process.

Thanks for the response!

 

[181228]

In follow-up to my previous posts 13 and 14:

I can confirm that the problem appears to be software related, and specific to 11.2. I have subsequently used the same USB sticks to perform installs of Centos 7, FreeNAS 11.1 and FreeNAS 9.10.2-U6 (561f0d7a1) without any problems. Those same USB sticks are now back in use as a mirrored boot pool for running 9.x and they are behaving appropriately ... just like they were before all of this misery began associated with the attempted upgrade to 11.2.

My FreeNAS server had previously been up and running on 9.x for many years without any issues. I had to rebuild the zpool to expand it's storage capacity, so I attempted to perform the data migration and a software migration to 11.2 while the system was down. The zpool migration went as planned, but the software migration evolved into a major headache that lasted for a solid week. Ultimately when there was no end in sight for the problems associated with migration to 11.2 I was forced to revert to 9.x as it was confirmed to be stable and reliable. At least I accomplished the zpool expansion as planned.

During the course of my week trying to get 11.2 to work I filed 4 different bug reports for 4 different issues. Unfortunately those but reports will probably end up going nowhere, as I had to move on because I can't commit the resources to testing in-development software. My hand was forced -- when my time budget ran out for the upgrade I had to revert to something that was known to work, and move on.

As it turns out I am now receiving responses to the bug reports where the devs are now asking for more information. I won't be able to assist them because I've had to move on. If any of you are enthusiasts, with systems that you can dedicate to testing, please file the bug reports and follow-up on them. There are some significant issues that escaped detection the testing phase and the devs are trying to work on them now. I just won't be able to help them this time around. Sorry.

 

[Meyers]

Just upgraded our production systems to 11.2-U1. Doesn't seem like this issue is happening in these systems. Same exactly USB drives but SuperMicro X10 series boards. Been running for the better part of a day and all seems fine.

On the backup server, I had to revert to 11.1-U6 because the system was unusable. It's already booting EFI but I could try booting legacy.

 

[averyfreeman]

Have you tried looking at your DMESG for errors?

 

[Meyers]

Of course. No errors anywhere.

 

[billias]

Adding on this Though not a server Motherboard (Asrock H370M-ITX | I5-8400) Does the same with USB3.0
I also have had this with different boards, and My DC rented server has these issues too! I asked DC explicitly to Attach USB device on USB2 port.
Poor performance under 11.2U(1|2|2.1)
It is the first Time I see my most "cheap" USB2 devices reading/writting faster than my USB3.0

Additionally any usb3 device, connected on USB3 port has issues, when the same connected to USB2.0 port performs better.

 

[Diode]

Thought I'd chime in. I have two systems running on stock HP DL380 G7 hardware with high quality mirrored USB sticks (verified not counterfeit) for the boot pool. I also have experienced extremely slow installs, very slow reboots with a cornucopia of errors (just about everything in this stream). I have not been able to do an online update for several revisions recently, and always have to do a clean install with every update (and hope that my main data pool stays un-corrupted). After a reboot, I always have to re-enter the IP address manually at the console (yet it always remembers my settings), then manually force an Active Directory update because that first try of that always fails too. Before 11.0 (or so) I had none of this trouble. I have no available drive bays, so I wonder how I might be able to go the SSD boot pool route in this kind of machine.

 

[ThreeDee]

I have no available drive bays, so I wonder how I might be able to go the SSD boot pool route in this kind of machine.

use one of these? https://www.amazon.com/Protronix-Optical-Drive-Universal-12-7mm/dp/B004XIU4T2

 

[billias]

use one of these? https://www.amazon.com/Protronix-Optical-Drive-Universal-12-7mm/dp/B004XIU4T2

And if you have no Free SATA/SAS ports?

In my point of view: The issue here is the USB3.0 driver reliability issues/performance issues.

 

[ThreeDee]

I am out of SATA ports so I have my 128GB SSD connected via a SATA to USB cable and plug into the internal USB 2.0 port on the bottom of my mobo inside my case. Or plug into whatever USB 2.0 port you have available.

 

[Diode]

use one of these? https://www.amazon.com/Protronix-Optical-Drive-Universal-12-7mm/dp/B004XIU4T2

Thanks, Very cool. I also found PCI-e cards that can house multiple mSATA SSDs so I can have redundancy.

 

[billias]

Thanks, Very cool. I also found PCI-e cards that can house multiple mSATA SSDs so I can have redundancy.

it really doesn't solve the issue. right?

 

[Diode]

I will try it and see. It seems the issue is getting away from USB and there may be more than one way to do that. I found that installation from DVD using an external USB DVD drive is OK, so I can give up my optical drive.

 

[Meyers]

I might try having the data center people put the USB drives into the 2.0 slots if that's possible. Otherwise I've given up on USB for this system (the backup system). What's weird is they work fine in our production systems. Same USB drive different chipset.

I ended up installing plain old FreeBSD on the data pool and will probably just configure this system by hand.

 

[Diode]

I installed a PCI-e card with dual 16 GB mSATA SSDs. 11.2 U2.1 Install from CD went smoothly. it recognized my existing ZFS filesystem on an 8 drive RAID OK, and import of setup database was cool except for replication tokens, which I had to re-do. Reboot is nice and fast. No errors reported. It appears that The FreeNAS folks really need to stop saying you can run OS on USB sticks. I was running good enterprise-class hardware and using legit good quality USB sticks and the performance just got worse and worse with each new release, particularly after they got rid of Grub.
https://www.amazon.ca/gp/product/B0142MR3JM/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1
https://www.amazon.ca/gp/product/B07783QX2C/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

 

[Diode]

Thanks, Very cool. I also found PCI-e cards that can house multiple mSATA SSDs so I can have redundancy.

Thanks for pointing me in the direction that solved my problems.

 

[Meyers]

I installed a PCI-e card

I'm glad that resolved it for you. I wish I had that option.

11.2 is where USB performance tanked for me. 11.1 still works fine on this system.

 

[Diode]

Might try a USB to mSATA adaptor with mSATA SSD in it?

 

[Meyers]

I'm not sure how that would help. That said, this is rented hardware we can't modify.