Advice on mirrored USB vs single SSD for boot

[Linkman]

Currently I'm using two 32GB USB flash drives in a mirrored config for booting. I will soon have a 60GB SSD available from another system, and wanted to ask the users with more experience than me, which would you prefer? Is a single SSD any advantage (speed, size?) over mirrored flash drives (redundancy?)? The SSD is basically free, and the USBs are already in use. Thanks.

 

[HoneyBadger]

Size isn't a huge issue as your existing USB devices are fairly large (32GB) but the SSD will be much, much faster to boot, write config, and especially ingest an update.

If you routinely back up your FreeNAS configuration to another location, I'd be inclined to go with the single SSD; but you do lose the "run-time redundancy"

 

[Ericloewe]

Single SSD, assuming no investment required (and that it's a good brand).

 

[brando56894]

I was using 2 USB 2.0 mirrored drives and then I switched over to a 4 GB SLC SATA DOM and there isn't really that much of a difference in boot time. There is a slight difference though in deleting things, such as the boot environments.

 

[Johnny Fartpants]

I would stick with your two usb sticks. At least youve got some resiliency if one was to die and its easy to replace with another stick so long as its 32G or bigger. Once the system has first booted it runs in RAM anyway so shouldnt make much if any difference.

 

[HardChargin]

I'm a fan of the SSD option myself, but only due to a "perceived" reliability improvement. Personally, I've tried both (single USB though) and had no issues with either. Nor did I notice any substantial speed increases in regular use with the SSD where I thought to myself, "wow, what an improvement".

 

[Nick2253]

I wouldn't change, though your needs don't necessarily match my needs. Here's the thought process I went through:

1. The USB system is already working. If it ain't broke, don't fix it. Couple points to USB.
   
2. Reliability isn't critical since I do regular config backups (you do regular config backups, right?), but mirrored drives are going to be more reliable than a single drive. Point to USB.
   
3. Speed isn't huge, since the OS is run from RAM once booted. I rarely reboot my system, and updates only happen occasionally, so the SSD benefits only matter for brief periods. Point to SSD.
4. 32GB is more than enough for FreeNAS, and 60GB doesn't really get you anything. Half point to SSD.

On the whole, I don't think either option is really bad, but I do lean towards the mirrored USBs.

 

[Darren Myers]

I picked up 2 used 80Gb SSD's for 20$ each off ebay, i much rather have an SSD then a USB drive for boot. http://www.ebay.com/itm/SSDSA2BW080...IES-2-5-SSD-/252056334460?hash=item3aafba707c

 

[cyberjock]

I heavily lean towards a single SSD versus USB, even 5 mirrors of USB. When you look at USB there are three basic problems that seem to hit 99% of users:

1. You buy some brand or model that just doesn't work at all or is horribly unreliable.
2. USB underperforms in terms of random write workload, and this can create I/O timeouts and other problems (especially when doing things like OS upgrades).
3. The USB chipset that your board uses just isn't very robust and/or compatible with FreeBSD and you have random problems that end up with a system that crashes or is otherwise unreliable.

#1 can be resolved by buying some other brand or model. The problem with trying to avoid #1 is that there's no 100% certainty that what you buy will or won't work until it is tried. Kingston used to work well, then suddenly they didn't. Same part number and same product. So you, as the end-user could never have anticipated that it suddenly wouldn't work.

#2 and 3 cannot be solved by simply "replacing the USB stick". So 2 of the 3 major "problems" you can can only be resolved by not using USB for your boot device.

Put all this together and you have these fairly 'unfixable' reasons to just not use USB for your boot device. Couple that with the fact that if your boot device(s) cannot function, your server will not function, and it's a common case of single point of failure for your server.

Yes, there is a very small subset of users that have USB sticks that "go bad". Here in the forums if anything sounds like a bad USB stick we just tell people to replace it. It's a cheap fix and is far easier to simply replace a $10 USB stick than spend lots of hours trying to prove if it was the case.

SSDs also last FAR longer than USB since they have wear leveling and other benefits, and you totally avoid #2 and #3 as a point for problems. So SSD (even single disk boot device) is far better (in my opinion) when considering all aspects of the problem, the consequences of the problem, and the relatively small cost to buy some small SSD to use as the boot device. One and done as far as I am concerned when you buy (or even reuse an old) SSD.

 

[balanga]

Just out of interest, asking as a newbie, after booting, when is the boot device used? How can I tell when it was last written to?

And, on average, how often does a FreeNAS system get rebooted?

 

[Nick2253]

My understanding is that the boot device is only written to when you do an OS update, or when you change the config. Otherwise, it just sits there.

As far as reboots go: I only reboot my server when I need to because of an update, or because I move my server. That puts reboots in the single digits annually.

 

[danb35]

Reliability isn't critical since I do regular config backups (you do regular config backups, right?), but mirrored drives are going to be more reliable than a single drive. Point to USB.

Mirrored drives are certainly going to be more reliable than a single drive of equal MTBF. However, if the MTBF of the single drive is orders of magnitude longer than that of the mirrored drives, this isn't the case. I haven't done the math to know where the break-even point is, but given the notorious crappiness of USB drives, it's something to consider.

Speed isn't huge, since the OS is run from RAM once booted.

I don't believe this is correct with 9.3. With 9.2 and earlier, the OS was loaded into a RAMdisk on boot, but in 9.3, it runs from the boot pool. ZFS caching means that most of it will live in the ARC most of the time, but not necessarily all the time. Write speed is a much bigger difference, but it is only really relevant when doing things like installing updates or deleting boot environments.

In my case, I'm using mirrored SATA DOMs, which is likely overkill to an extreme degree, but it seems to work well.

 

[joeschmuck]

And of course I'll chime in here...

It depends on how many available SATA ports you have but assuming you have a port available so the correct answer is SSD. Many folks made great points above, for me it's the longevity and reliability of a SSD over a USB Flash drive.

This does not replace not having a current backup of your configuration file so if you don't already do this the smartest thing you could do is to create a backup of your configuration file (some folks do this nightly) and have it available, meaning don't back it up to your NAS, that is an idiots move. This is so you could recover with ease should you need to. I can't tell you how many times we have read a thread where a person had a USB flash drive die and they cannot recover easily because they never made a backup of the configuration file.

Lastly, since the SSD is used, I have no idea how much life is left on it, use something like CrystalDisk or read the SMART data to see how much life remains on the SSD.

 

[cyberjock]

Mirrored drives are certainly going to be more reliable than a single drive of equal MTBF. However, if the MTBF of the single drive is orders of magnitude longer than that of the mirrored drives, this isn't the case. I haven't done the math to know where the break-even point is, but given the notorious crappiness of USB drives, it's something to consider.

That would be true, if everything else was equal. However, everything is most certainly not equal. Read my response about how USB kind of sucks and you'll see why USB, even in a mirror, can be far far worse than an SSD. If you don't believe it, feel free to come over to my house. I have a system that will not work properly with USB, even if I had 4xUSBs in a mirror, but works perfectly with an SSD.

I don't believe this is correct with 9.3. With 9.2 and earlier, the OS was loaded into a RAMdisk on boot, but in 9.3, it runs from the boot pool. ZFS caching means that most of it will live in the ARC most of the time, but not necessarily all the time. Write speed is a much bigger difference, but it is only really relevant when doing things like installing updates or deleting boot environments.

Memory devices are still used in 9.3. Also, ZFS caches stuff in RAM, so the stuff you use alot is still cached in the ARC.

 

[danb35]

That would be true, if everything else was equal. However, everything is most certainly not equal.

Also, ZFS caches stuff in RAM, so the stuff you use alot is still cached in the ARC.

I can't tell if you're agreeing or disagreeing with me. Your tone sounds to me like you're disagreeing, but you're then proceeding to restate what you quote me as saying.

 

[joeschmuck]

I can't tell if you're agreeing or disagreeing with me. Your tone sounds to me like you're disagreeing, but you're then proceeding to restate what you quote me as saying.

He's disagreeing with you about the reliability of mirrored devices.

Mirrored boot devices is a joke and is something of a sales tactic to me. I didn't want to comment on it but after Cyberjock jumped in, well...

Honestly, not all motherboards will properly support this so called automatic failover. Typically when you setup your MB you tell it which device is the boot device which means you likely have to pick one of the USB devices, or SATA drives and place things in a boot order sequence. So lets just say that works fine, now you have a primary boot device and at some point in time it fails. The question is, do you really think it will fail and not lock up your machine? Seriously, only proper intelligent hardware can pull off something like that. Here is a perfect example... Your normal boot flash starts the bootup process and then crashes about half way into it with some fatal trap. How in the world will the mirrored device magically fix this? I'm all ears for a realistic answer that is completely free of operator interaction.

Seriously, operation interaction is needed and once you have figured out the failed flash drive, you remove it and possibly reconfigure the machine BIOS to get things running again. To test the failover you just cannot just pull one of the flash drives out, you need to have a no shit failure. This is why I don't like mirrored boot devices, it's false hope in my mind. Backup the configuration file and be ready to rebuild the boot device if needed. When it comes to a SSD, odds are in favor you will not have a failed boot device due to a hardware failure of the SSD. You can bet that you will have a USB flash drive failure eventually, they have a finite life span.

So we are back to a single boot device, SSD is my preference and I'll get there after FreeNAS 10 hits the streets because there is no reason to upset my system while it's working fine. Also I'll have to blow the dust bunnies out when that time comes so I'll have to open the case anyway.

At the OP, I hope you post what you end up doing.

 

[Ericloewe]

Mirrored boot devices is a joke and is something of a sales tactic to me.

I tend to view them as somewhat more useful than that.

The bootstrap process is indeed iffy. A ZFS extension for UEFI would be nice, but motherboards would need a smallish flash device for this stuff.

Once GRUB is loaded, though, the ZFS mirrors are used as such. Since GRUB itself is a smallish (set of?) executable(s?), that means that many failure modes can be corrected from that point on. Of course, it does not solve the problem of completely borked devices.
This could *probably* be solved with a third device, small but reliable, which is infrequently updated and would contain GRUB. This more reliable device can thus successfully load GRUB in a wider range of circumstances. It sounds silly and it is rather silly.

Seriously, operation interaction is needed

Totally agreed, but I wouldn't call it a negative aspect, since the alternative suffers from it too. I'd call it a "lack of positive aspect".

Of course, the way to avoid Wile E. Coyote-esque contraptions is to use an SSD - those are reliable enough. Since outright failure is less likely with SSDs, mirrored boot SSDs would be primarily to correct corruption. Whether that is really needed in a boot device (i.e. detection isn't enough) depends on the application.

 

[rogerh]

Am I right in thinking that we still can't replace a mirrored boot device that has failed and expect the new device to be bootable unless we manually install grub via the CLI on the new device? Presumably this applies to SSDs as well as USB drives?

 

[Ericloewe]

Am I right in thinking that we still can't replace a mirrored boot device that has failed and expect the new device to be bootable unless we manually install grub via the CLI on the new device? Presumably this applies to SSDs as well as USB drives?

The situation is convoluted enough that jkh asked in a bug ticket the other day and no answer has been given yet.

Adding a device seems to work correctly, supposedly, though.

https://bugs.freenas.org/issues/6993

 

[rogerh]

The situation is convoluted enough that jkh asked in a bug ticket the other day and no answer has been given yet.

Adding a device seems to work correctly, supposedly, though.

https://bugs.freenas.org/issues/6993

I wonder if it would work to add a third mirror device and then remove the defective one?