Fresh install 11.2 on USB drives, terrible performance

[Diode]

I'm not sure how that would help. That said, this is rented hardware we can't modify.

These things can be plugged into external USB jacks, so they are like a big wide USB stick. Haven't tried one in a FreeNAS system, but they do work, often much faster than USB sticks. https://www.amazon.ca/USB-mSATA-Adapter-Disk-Driver/dp/B00CJE7BRA

 

[Meyers]

Ahh well that won't help if this is an issue with the chipset / drivers though. :(

 

[pschatz100]

There are a couple of people that report they have had some success using USB with external SSD. Next time I rebuild a system, I'm going to try that approach. If it works, it will free up a SATA port for storage.

 

[dashtesla]

I haven't tried anything with USB yet but i did try both SAS drives through hba mirror and also another server i ended up just using a p400 raid6 volume because of controller /backplane layout limitations/convenience, both worked really all of them just some old 146GB hp drives i already had both cases the ui is very snappy and responsive boot times are trivial. I will try later some usb controllers I have with sata drives and ssds when i get around to it i'll share results.

 

[Sean Ogle]

I've had this problem now on two different servers:

1. HP Microserver Gen 10
2. Dell R510

I've replaced the RAID cards with HBA capable cards in both systems (a real pain on the HP!). BTW, setting up FreeNAS on this server was an odyssey in and of itself.

On the HP I finally caved in and bought an internal SSD. That system boots fine now.

On the Dell system I was able to adjust the BIOS setting for the USB stick emulation. Initially it was "AUTO", which resulted in the eternal boot times (and faults during the upgrade from 11.2-U5 to 11.2-U6). Now I've switched it to "Hard disk", and booting is back to normal. BTW, the third USB stick emulation option is "Floppy". So it seems the slowdown occurs in 11.2 when the BIOS is configured to have the USB stick emulate a floppy (AUTO must be choosing "Floppy" instead of "Hard disk", although I did not experiment with using "Floppy" directly).

One thing to note is that the Dell Poweredge R510 doesn't even have USB3, only USB2. So I don't think this is a USB2 vs. USB3 issue. The HP has both USB2 and USB3 ports, but both resulted in pathetic boot performance.

A previous poster mentioned something similar.

I hope this helps somebody in the future.

 

[dhenzler]

SOLVED

Posted on March 3, 2019 by Daniel Mettler
FreeNAS 11.2/FreeBSD BTX boot loader 1.0: Very slow booting if “Legacy USB 3.0 Support” is enabled

Starting with version 11.2-RELEASE-U1, FreeNAS uses the FreeBSD BTX 1.0 boot loader instead of Grub (reason: The FreeBSD boot loader finally started supporting ZFS – years after Grub supported it, BTW).
Unlike Grub, the FreeBSD boot loader seems to have rather “suboptimal” support for storage devices (like USB sticks, USB drives) that are accessed using the legacy USB 3.0 mode at boot time. So, when I upgraded FreeNAS from 11.1-U7 to 11.2-RELEASE-U2 on my home-built ASRock FreeNAS E3C226D2I NAS appliance, booting FreeNAS from an internal 16 GB USB 3 stick took all of a sudden almost 30 minutes(!) instead of the usual 1-2 minutes.
As I found out by “trial & error” (or educated guessing, as some would say), the only reason for this huge slowdown was that “Legacy USB 3.0 Support” was enabled in the AMI BIOS (the effect is reproducible). So, just disable “Legacy USB 3.0 Support”[1] in the BIOS and FreeNAS >= 11.2 will boot quickly again.
[1] Note: There’s no need to disable “Legacy USB Support”, so you’ll still be able to use your USB 2 keyboard or mouse at boot time.

 

[mgd]

Is there any way to work around this problem on ASRock Rack C2550D4I? I have been looking through the BIOS options and could not find any usable options to tweak here to make the USB boot to FreeNAS 11.2-U7 fast.

If not, are my only options these?

1. Downgrade to 11.1
2. Buy SSD boot devices

BTW: It seems this thread I started is really just this problem.

SOLVED - Very slow boot and shutdown (and reboot) after upgrade to 11.2-U7 from 11.1

EDIT: See the conclusion at the bottom of this post with instructions. Quick summary: 11.2 requires more from its boot devices compared to 11.1. If you experience very long boot times after upgrading to 11.2, then replace your boot devices with SSD drives. --- ORIGINAL QUESTION I have a...

www.ixsystems.com

 

[pschatz100]

You might try creating a boot device for Legacy (sometimes also called BIOS boot.) Oftentimes, older motherboards are a little fussy about the boot devices - some devices will work better when set up for UEFI, others will work better for BIOS boot. I had this happen with a Supermicro X9SCM. Solved my problem by re-configuring the boot device.

 

[mgd]

Sounds like a good idea to try out.

Actually, I thought of trying:

1. Create a new BIOS / Legacy boot device and see how it boots.
2. If that was fast, create a new UEFI boot device and see how it boots.
3. Using my existing boot devices (mirrored) boot into my previous 11.1 environment and see how it works.

If #1 is fast and #2 is slow, it points to a problem with UEFI / BIOS FreeBSD bootloader incompatibility.

If #3 boots fast (into 11.1), I would suspect the 11.2 kernel to have a problem with the USB to SCSI driver.

What puzzles me is that the problem is not initially at boot (which could point to UEFI / BIOS FreeBSD bootloader incompatibility) but first when the kernel tries to mount the root filesystem:

Trying to mount root from zfs:freenas-boot/ROOT/11.2-U7

which points to the kernel driver support for USB.

Is it correct to assume my current boot devices are UEFI if it looks like this?

Code:

$ gpart list
...
Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 30273566
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   efimedia: HD(1,GPT,fa3671af-7d20-11e6-9133-d05099c0ea6d,0x22,0x400)
   rawuuid: fa3671af-7d20-11e6-9133-d05099c0ea6d
   rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
   label: (null)
   length: 524288
   offset: 17408
   type: freebsd-boot
   index: 1
   end: 1057
   start: 34
2. Name: da0p2
   Mediasize: 15499517952 (14G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 544768
   Mode: r1w1e1
   efimedia: HD(2,GPT,fafa66bd-7d20-11e6-9133-d05099c0ea6d,0x428,0x1cdebf0)
   rawuuid: fafa66bd-7d20-11e6-9133-d05099c0ea6d
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 15499517952
   offset: 544768
   type: freebsd-zfs
   index: 2
   end: 30273559
   start: 1064
Consumers:
1. Name: da0
   Mediasize: 15500083200 (14G)
   Sectorsize: 512
   Mode: r1w1e2
...

 

[pschatz100]

I cannot say, from what you posted. When you create the boot device from the installation image, don't you have the option to create either a BIOS boot device or a UEFI boot device. At least, it did the last time I did a fresh install.

Sounds like a good idea to try out.

Actually, I thought of trying:

1. Create a new BIOS / Legacy boot device and see how it boots.
2. If that was fast, create a new UEFI boot device and see how it boots.
3. Using my existing boot devices (mirrored) boot into my previous 11.1 environment and see how it works.

If #1 is fast and #2 is slow, it points to a problem with UEFI / BIOS FreeBSD bootloader incompatibility.

If #3 boots fast (into 11.1), I would suspect the 11.2 kernel to have a problem with the USB to SCSI driver.

I suppose you could go through all the trouble to figure this out... I never bothered. Initially, I used to boot from mirrored flash drives. When I updated to FreeNAS 9.10, I decided to change to an SSD. My flash drives had been created for UEFI boot, and when I first used the SSD I created it for UEFI boot as well. The system never started properly. After a lot of troubleshooting (which did not turn up any hardware problems), I tried setting up the SSD for BIOS boot and bingo!!, it all worked - and has been stable and trouble free since then. I never looked back. At the time, some folks commented that UEFI boot was a little touchy in the early days.

And, I don't think you can be certain about your assumption #3, because they changed how FreeNAS boots with 11.2. The problem could be caused by a small incompatibility between the firmware on the particular boot device and the motherboard. For an old motherboard, I'm not certain it is worth the trouble to figure this out.

Never forget that FreeNAS is built upon FreeBSD - which is not Windows. Just because something works on Wndows is no guarantee that it will work in FreeNAS. Hardware manufacturers test to make certain their products run on Windows, but they don't all test for compatibility with FreeBSD. In some cases, they implement Windows drivers that have bug fixes for device firmware bugs.

 

[mgd]

I cannot say, from what you posted. When you create the boot device from the installation image, don't you have the option to create either a BIOS boot device or a UEFI boot device. At least, it did the last time I did a fresh install.

From this line I assume the boot is UEFI:
efimedia: HD(2,GPT,fafa66bd-7d20-11e6-9133-d05099c0ea6d,0x428,0x1cdebf0)

But I am not certain, so I will try to create both a BIOS and UEFI media – and, yes you are right. this can be chosen in the installer.

Never forget that FreeNAS is built upon FreeBSD - which is not Windows. Just because something works on Wndows is no guarantee that it will work in FreeNAS. Hardware manufacturers test to make certain their products run on Windows, but they don't all test for compatibility with FreeBSD. In some cases, they implement Windows drivers that have bug fixes for device firmware bugs.

I am aware of that. (Personally, I mostly run macOS and Linux.)

And, I don't think you can be certain about your assumption #3, because they changed how FreeNAS boots with 11.2. The problem could be caused by a small incompatibility between the firmware on the particular boot device and the motherboard. For an old motherboard, I'm not certain it is worth the trouble to figure this out.

My idea is that if booting up with the new bootloader (11.2 uses the FreeBSD bootloader and not GRUB as in previous releases of FreeNAS) in the old 11.1 profile works fast, then it is not the bootloader that is causing the problem but instead which drivers were compiled into the FreeBSD kernel by the FreeNAS team and I would then point to the driver that emulates USB flash devices as SCSI / SATA disks to the OS as having a problem.

 

[mgd]

Ok, an update on this.

Short version
It seems that FreeNAS 11.2 requires much faster boot device than FreeNAS 11.1 did so I will switch to SSDs on SATA for boot.

I will probably go for Kingston SSDnow A400 in the 120GB version and have 2 of those for a mirrored boot device, do a fresh install of FreeNAS 11.2, and then reimport my configuration and storage.

It also does not seem to matter (for me, on the ASRock Rack C2550D4I motherboard) whether the boot device is BIOS or UEFI.

And BTW, my existing boot device is actually a BIOS boot device (and not UEFI as I incorrectly suggested above). If it was a UEFI boot device, the first GPT partition would have exactly this rawtype (output from gpart list):
rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
which is has not.

Long version
After shutting down the system, I disconnected all my storage disks and my boot devices (2 x Kingston DataTraveller 2, 16GB, USB2). I then borrowed two USB sticks and used one for the installation media and the other as a new boot device to try out a fresh install.

The boot device is a Kingston DataTraveller 3, 64 GB, USB3. However, the C2550D4I motherboard has only USB2 ports, so it will be used as a USB2 device.
The USB device is mapped to SCSI with this driver in FreeBSD (as seen on the console output):
Removable Direct Access SPC-4 SCSI device

1) Fresh 11.2-U7 installation with BIOS bootloader
Installation: 12:30 minutes.
First boot: 7 minutes. (first boot includes some first time initialization and is a little slower)
Second boot: 6:30 minutes.
Mounting root filesystem: 1-2 minutes
Shutdown and poweroff: 25 seconds.

2) Fresh 11.2-U7 installation with UEFI bootloader
Installation: 9:30 minutes
First boot: 7:30 minutes.
Second boot: 6 minutes.
Mounting root filesystem: 1-2 minutes
Shutdown and poweroff: 10 seconds.


Now, I removed the boot device and reattached my existing mirrored boot devices (which are BIOS, see above), but left my storage disconnected.

The USB device is mapped to SCSI with this driver in FreeBSD (as seen on the console output):
Removable Direct Access SPC-2 SCSI device

3) Boot into 11.1-U7 environment on existing boot media – storage disks disconnected
Boot time: 4:30 minutes
Mounting root filesystem: 1:30 minutes

4) Boot into 11.2-U7 environment on existing boot media – storage disks disconnected
Boot time: 14:30 minutes – yes, 10 minutes slower.
Mounting root filesystem: 2:30 minutes


Now, I reconnected my storage disks.

5) Boot into 11.2-U7 environment on existing boot media – storage disks connected
Boot time: 24 minutes – yes, an additional 10 minutes slower
Mounting root filesystem: 7 minutes

Conclusion
To me it simply looks like FreeNAS 11.2 requires a much faster boot device compared to 11.1. I am not sure why.

I also do not understand why mounting the root device with storage attached takes so much longer than without storage attached. The reason might be that it actually does not take long time, but that FreeBSD/FreeNAS does not output messages of what it is doing right after mounting the root filesystem so it is something else (involving reading and/or writing to the boot media) that takes the extra time.

So the conclusion must be that it is time to move to SSDs for boot device.

I will probably go for Kingston SSDnow A400 in the 120GB version and have 2 of those for a mirrored boot device, do a fresh install of FreeNAS 11.2, and then reimport my configuration and storage.

 

[mgd]

I got my two new SSD drives today, and after installing them as mirrored boot device a complete boot measured from after UEFI POST and until the system is completely ready now takes less than 2.5 minute.

When installing the SSDs I detached one USB boot drive and attached one SSD at a time to the boot pool. Resilvering from the remaining USB drive to the first SSD took almost 4 hours and resilvering from the 1st SSD to the 2nd SSD took 2 minutes and 18 seconds, so I guess the USB drives have been close to dying and combined with 11.2 reading/writing more to the boot drive this has caused the problems for me.

 

[dfad]

Joining the club.
With FreeNAS 11.3-U2.1, and also with a pair of Kingston DataTraveller 3, 64 GB as boot device, like topic starter had. Both drives are new and work brilliant outside of FreeNAS, for example if used as Ubuntu Installer drive.
Both in UEFI and in BIOS mode it's about 1.5-2 h to install FreeNAS, with very slow updates on 'progress bar', and 0.5 h to boot. Just to check, I tried to install FreeNAS on SATA HDD connected to same system - it works fine, installation is just few minutes.

 

[dhenzler]

Greetings,

We rented a new server to be used for backups. This server has this motherboard and we're using two Kingston Data Traveler 3.0 64GB sticks which are plugged into the USB 3.0 ports I believe. This is a setup we've been running for over a year on different Supermicro hardware without issue.

I installed a fresh copy of 11.2 on this system using both USB drives. Initial boot takes 30 minutes or more. Any configuration attempt at all locks the SQLite databases, causes UI timeouts, etc. Performance is so terrible as to make this setup completely unusable. We've had them replace our USB drives and have done several fresh installs and we get the same result every time. I've tried installing on just one USB drive and get the same results as well.

I booted SystemRescueCD and formatted these drives as XFS and ran a few basic tests i.e. dd if=/dev/zero if=test.bin bs=1024000 count=16000 (this system only has 16GB of ram). Performance was what I'd expect from a USB stick. No hangups, no 3s+ response times. It seems like for whatever reason this is a FreeNAS issue.

There are no log entries that I can see such as hardware timeouts. No other log entries other than the various middleware processes complaining about locked DB and timeouts.

Does anyone have any idea what I could do to troubleshoot this further? I'm tempted to say the heck with it and just install the OS on the data drives, which work just fine as far as I know. But this isn't a rush so I'd really like to figure out what's going on if possible.

Thanks.

 

[dhenzler]

According to the changelog for 11.2-RELEASE, the boot process was changed - so there could definitely be an issue with thumb-drives. If I were you, I would try two things: 1) try re-creating the boot drive while it is plugged into a usb2 port. Don't create mirrored boot drives - just create a single drive and see if that makes any difference, and 2) try a different brand thumb-drive (a different drive will have a different controller).

When I used to boot off of USB flash drives, I always used Sandisk thumb-drives. Often, one went bad when I tried to update FreeNAS. Sometimes, I could take the bad thumb-drive and reformat it for windows - but usually not. After I switched to booting off a small SSD, all these issues went away. I did not look into the matter any further once I started using an SSD for boot.

I'm using a Patriot 8GB thumb drive (just one) to boot up my HP Proliant DL380e Gen8 and it works like a champ. I have another using a different Thumb drive and it works great as well. Both machines are using Seagate ES2 or ES3 SAS drives for storage. One having a dozen 3TB drives and the other with five 3TB drives. System performance seems tip-top to me... I put the boot drive INSIDE the case on the motherboard in a socket meant for boot drive. I also have a SATA boot disk in another DL380p Gen8 and notice no difference in boot performance... other than speed... I used it because of an experience with a IBM/Xyratex HS-1235 which had a USB conflict causing extremely poor USB boot speed. Perhaps your boot issues are USB related!

 

[PDM]

I never had any joy with the "Kingston DataTraveler 3, 64 GB (DT50)" usb sticks. Had to revert to Kingston DataTraveler 2, 32GB. :(