3-way boot mirror is not supported in 11.1-U7?

JustinOtherBobo

Dabbler
Joined
Aug 21, 2018
Messages
26
To my surprise I discovered today that 3-way boot mirrors are not supported in 11.1-U7 Exception Type: MiddlewareError at /system/bootenv/pool/attach/ Exception Value: [MiddlewareError: [EFAULT] 3-way mirror not supported yet]
Booting off USB is useful because it uses "free" external USB connections.
With a lot of existing hardware, anything else ends up using connections that could be assigned to storage pool resources instead.

Even though USB sticks are ridiculously crappy and short lived, they are also ridiculously cheap, and many systems have a multitude of USB 2.0, or higher, connections sitting un-used.

Absent a particularly tricky bit of programming, why limit boot sticks to 2-way mirrors when increasing boot pool redundancy to 3-way or 4-way or higher would ensure that scrub has enough information to correct errors even under extremely unlikely conditions-while the incremental hardware expense is at the level of a cup of coffee?

Is this supported with later releases? Did it use to be supported and it no longer is?

Thank you and my apologies if this has been addressed in the past, or should be in a different forum location!
 
Last edited:

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
... I have been told that 3-way boot mirrors were an option in the past. I ran into this issue earlier this year. As of 11.2 U3, a 3-way mirror was not possible. That may have been fixed in the meantime, but I have no way to test at the moment.
 

JustinOtherBobo

Dabbler
Joined
Aug 21, 2018
Messages
26
hmmm, I suppose it must fall under the tricky programming bit?

Otherwise given the cost of an 8 or 16GB stick (or even 32) it seems silly not to to pile them on into an n-way mirror that ought to ensure resiliency.

As mentioned, I don't want to use an internal connector since that may impact on maximizing storage.

Just got an email about another USB boot stick getting kicked out for too many errors :(
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Given the strain that FreeNAS seems to put on its boot media, I don't quite understand why USB flash sticks are still still supported. At the very least, issue a warning to use high-endurance flash media, i.e. something based on SLC, I guess. Perhaps a high endurance SDXC like the Samsung Pro, combined with a USB adapter? That's the kind of boot media I use for my Raspberry Pi's.

SuperMicro claims my SATADOMs are good for 1 write per day... which is about 64GB. Seems unlikely my FreeNAS will ever get close to that... and yet I still have a mirrored boot pool.
 

macatarere

Dabbler
Joined
Jun 11, 2019
Messages
17
A StarTech sata to usb adapter with ASMedia - ASM1153E chipset is booting 11.3-BETA1 on an X8SIE that will otherwise not install or boot from any USB drive I've tried. The SDXC won't need a drive bay, anyone tried it?
 

JustinOtherBobo

Dabbler
Joined
Aug 21, 2018
Messages
26
A couple of good ideas have come up in this thread and which I am trying them, generally successfully, on my 11.1-U7 versions running on HP Z800 workstations using SATA II 3.0b/s interfaces internally and USB 2.0 ports externally and generally just storing backups and replicating back and forth :) My use case is to avoide using "scarce" internal SATA ports to boot from since I reserve them for my storage and transfer hard drives.

-- after a 16GB Lexar USB 2.0 stick failure, I grabbed the best sounding name brand 32GB SDHC I run into Walmart--will have to track the receipt for brand/model--and a BlackWeb 4:1 kit with 64GB SDxC plus Micro SD and SDxC to USB A adapters so as to use the adapter. The SDHC and adapter combo were recognized on insertion and the mirror resilvered and all was good!

Just in time as within a day or two of installation the 16GB USB 3.0 pair in the original mirror hard crashed with both read and write errors.

-- got an inexpensive (at this time) Apacer Panther AS340 2.5" 120GB SATA III SSD and a nicely packaged but not terribly impressively constructed NexStar TX (Vantec) SuperSpeed USB 3.0 2.5" SATA enclosure from a supplier I deal with regularly. Plugged it into the USB 2.0. Was NOT recognized and made available using 11.1-U7 in the HP Z800 I tried it with. I rebooted and it WAS recognized. The mirror re-silvered and the system says all is good.

For further testing will be replacing the 32GB SDHC in adapter with a second Panther AS340 / Vantec combo and checking if it will boot, as I've had a past incident where a resilvered and reportedly fully restored mirror would not boot from the new drive when the original mirror member subsequently failed.

Would love to know if anyone knows of any 2.5" UASP and/or 3.1 enclosures... if I am buying a dozen enclosures they might as well be slightly better ones for future use... as long as they also work today!

Oh, and I ordered a couple of Samsung Pro Endurance 64GB for testing (they are speced to last longer than the 32GB Pro Endurance for a marginal cost increase from the source I found them at) but I am still looking for an SDXC to USB A adapter for them other than buying the same BlackWeb kit.

One concern in all this would be power "saving" features that may be built into any of the above...

Thoughts?
 

macatarere

Dabbler
Joined
Jun 11, 2019
Messages
17
I’m about to try a Kingston SDXC setup. No failures yet with Samsung BAR Plus, 5 months in, 5 year warranty. Would a spare be better than a 3 way mirror? No wear until it’s needed.

The microSDXC is booting in a Kingston USB A adapter. The adapter has two slots, will try two cards. Two adapters four cards?

I'm seeing a lot of USB boot failures with 11.3-BETA1 in an X8SIE, especially mirrors, the type of media doesn't make much difference. Unplugging one of the mirrors may get a boot.

This leaves me reluctant to put USB boot into production without a lot of testing first and recovery options ready. A board with extra sata ports could be a lot less trouble, 4-port SAS cards are going for as little as USD20.00.
 
Last edited:

JustinOtherBobo

Dabbler
Joined
Aug 21, 2018
Messages
26
The spare idea sounds good as it would avoid wear and tear. Plus given that many USB sticks seem to last at least SOME time (I do move any logging that I can to the main pool using the interface options), I would think that a 64GB Samsung Pro Endurance may have enough legs compared to the random Lexar/Kingston/SanDisk/Apacer/ADATA/TeamGroup/Blackweb 8/16/32GB USBs which have used at various times.

However, if the USB 3.0 or better SSD enclosure works reliably enough, the current cost of sub 128GB SSDs is not prohibitive compared to having to add an attached hot spare USB, or two, just "in case" or, worse, having to schedule travel to an ailing box just to attach a USB stick.

As to the (not) booting from the mirror, I need to test further to see under what conditions it happens. I hazily remember it happening twice but I was more concerned with booting the systems in question as opposed to investigating why exactly they weren't booting, though it is now in the back of mind to test that a new mirror member can boot if left as the only working drive!
 

macatarere

Dabbler
Joined
Jun 11, 2019
Messages
17
Three successful 11.3-BETA1 installs to X8SIE with BAR Plus 3.1 32/64/128GB but not for long, they all stalled at the Booting... message. Installed to a mirror of WD 2.5" HD's, connected to sata ports, they soon failed to boot too. 11.2-U7 is booting reliably on the same board with a mirror of Cruzer Blade 2.0.

The sata and usb ports connect to the same Ibex Peak 3420 chipset, I'm guessing this is an 11.3 3420 chipset issue?
 

JustinOtherBobo

Dabbler
Joined
Aug 21, 2018
Messages
26
Three successful 11.3-BETA1 installs to X8SIE with BAR Plus 3.1 32/64/128GB but not for long, they all stalled at the Booting... message. Installed to a mirror of WD 2.5" HD's, connected to sata ports, they soon failed to boot too. 11.2-U7 is booting reliably on the same board with a mirror of Cruzer Blade 2.0.

The sata and usb ports connect to the same Ibex Peak 3420 chipset, I'm guessing this is an 11.3 3420 chipset issue?

So you're saying that a fresh FreeNAS install to an external USB thumb drive connected to the system via USB and to a second install to a mirror of SATA connected 2.5" drives ALL fail to boot after install on an Ibex Peak 3420 chipset mainboard?

I have not dabbled with any of the Betas -- but I would assume that there is reporting that might be needed?!?
 

macatarere

Dabbler
Joined
Jun 11, 2019
Messages
17
Yes. Had I’ve known I would have tried the 2.5” WD’s sooner, a week of being so close to a reliable USB boot. An Intel S3420GP will USB boot but only BAR Plus 3.1. I’m assuming its not as marginal as the X8SIE. Debug files sent. I’ll be sure to keep beta away from production.
 

macatarere

Dabbler
Joined
Jun 11, 2019
Messages
17
After the boot failures of the last week it seemed a good idea to migrate a soon to be in production X8SIE from Cruzer Blade 2.0 to BAR Plus 3.1. The first resilver completed ok but the second crashed the system, a second attempt went the same way. Instead of a fresh install from USB/DVD media, an already installed but not configured drive was booted then resilvered to a mirror of BAR Plus 32GB, followed by a config upload. Although previous boot environments are no longer available this process was much quicker and boot times are significantly reduced.

My plan is to extend the idea of a spare to cabling the unused motherboard USB headers to a pci bracket for a spare plus unconfigured boot drive. No need to go hunting for installation media when it's needed most. It looks like the spare will only boot if it is the only boot drive plugged in, if other boot drives are plugged in, boot stalls.
 
Last edited:

NoTHX

Cadet
Joined
Jun 18, 2018
Messages
2
To my surprise I discovered today that 3-way boot mirrors are not supported in 11.1-U7 Exception Type: MiddlewareError at /system/bootenv/pool/attach/ Exception Value: [MiddlewareError: [EFAULT] 3-way mirror not supported yet]
Booting off USB is useful because it uses "free" external USB connections.
With a lot of existing hardware, anything else ends up using connections that could be assigned to storage pool resources instead.

Even though USB sticks are ridiculously crappy and short lived, they are also ridiculously cheap, and many systems have a multitude of USB 2.0, or higher, connections sitting un-used.

Absent a particularly tricky bit of programming, why limit boot sticks to 2-way mirrors when increasing boot pool redundancy to 3-way or 4-way or higher would ensure that scrub has enough information to correct errors even under extremely unlikely conditions-while the incremental hardware expense is at the level of a cup of coffee?

Recently I run into this problem as well. Feel free to prove me otherwise, but I do not see any reason why it should not work, in fact it does work, only the GUI refuses to do it. So let's help it a little over SSH.

GUI part: Let's say you have da0 and da1 USB drives setup as mirrors for freenas-boot pool in a regular way, and da2 ready for 3way mirror. From the GUI, click detach at da1 and at freenas-boot click attach da2, wait for resilvering to complete. You now have 2way mirror from da0 and da2.

SSH part: zpool attach freenas-boot da0p2 da1p2
Refresh the GUI, now you see 3way mirror in the resilvering process, with no objections from the GUI anymore :)

You can repeat the process how many times you want, fill all your USB ports with flash drives if you wish, provided all these drives were a "regular" member of the 2way mirror before (so it is properly setup by freenas - bootcode, etc... also you are sure this way the drive has proper size and could be used at all!) and then all Nth+2 attached one by one over SSH.

Regarding the reliability of the USB drives, I have no problem with Sandisk Ultra Flair 64GB drives so far, assuming there is plenty of space to distribute writes among. Also I do not use DC, and system dataset is of course off these drives, so the number of writes should not be too much. With fear of the reliability in mind, you may slightly adapt the process above and use hotspares (otherwise discouraged for data arrays) with zpool attach freenas-boot spare daXp2. This way no strain is put onto the USB drive, until you need it ;) 3way mirror and some spares in reserves should be safe enough to retain resiliency for almost all possible situations which could happen. To be honest I did not test the hotspares scenario (so it may have some unexpected side effects), but online mirrors work fine for me.

Of course this is not a replacement of backup, you always should have backup of the recent configuration aside.
 

artlessknave

Wizard
Joined
Oct 29, 2016
Messages
1,506
it's probably not "officially" supporte, but you can still create 3-day boot mirrors during install or at the commandline. it is only the GUI that has it disabled for attaching, you can still replace or detach them. I don't think the removal of n-way mirror on boot drive was needed, can't find any reason for it other than "We aren't doing it", particularly since the GUI already works with it.

at the commandline, you just create gpt partitions on your 3rd (should work for virtually any number really), attach to boot pool, and run a command to copy the boot code (conveniently, when you attach the disk to the boot pool, it tells you the boot code command)

i see what look like some ideas around using external USB RAID enclosures....the normal "do not zfs + RAID" would apply, particularly since USB enclosures are notoriously unreliable. you would be safer with just 2 way mirror tbh.

sandisk cruzer fit are popular choices - sandisk in general are more likely to have the wear leveling of SSD's on their usb drives, other OEMs leave this out as its cheaper, but uneven leveling makes the drives die drastically faster. I had some kingstons and they lasted about 8months, the cruzers are still going strong. the cruzers are also perfectly sized.

if you get any "SSD USB", you should find those also far more reliable than regular USB, since, again, SSD wear leveling should be present.
 
Top