Mirrored Boot Drive

[rmccullough]

I currently have 1 32GB SATADOM as my boot drive. I would like to add a mirrored backup boot drive.

I have seen several recommendations to go with a cheap SSD, but my enclosure does not have a spot to mount one conveniently.

So I am left with pursuing a USB stick or a 2nd SATADOM.

Can I mix SATADOM/USB Stick for boot devices and still mirror them? Is my best option to just purchase a 2nd SATADOM? This one seems reasonable, but not sure how good of quality it is. Or am I better off getting a matching SuperMicro 32GB SATADOM for just a few dollars more?

Is there a spot in the documentation explaining how to add a 2nd mirrored boot drive after the system has initially been setup?

 

[joeschmuck]

I have seen several recommendations to go with a cheap SSD, but my enclosure does not have a spot to mount one conveniently.

You could use zipties or even drill two small holes if you really needed to so you can screw it anywhere in the case. The SSD can be mounted anywhere that the cables reach. Think outside the box.

I don't think you need to mirror the SATADOM, just keep a backup copy of your configuration file handy, or you can pull the config file from your pool since it's backed up there every night. I prefer to just keep a copy of my own, it makes restoration just that much quicker.

Just remember, a mirrored boot drive does not mean there is an automatic failover, you would need a true RAID card to make that happen, and I mean that is would be an extra RAID card.

Can I mix SATADOM/USB Stick for boot devices and still mirror them?

I don't see why not, just remember my words above, it's not an automatic failover.

 

[Ender117]

Just remember, a mirrored boot drive does not mean there is an automatic failover, you would need a true RAID card to make that happen, and I mean that is would be an extra RAID card.

Ehhh Does this mean the system will halt if one in the mirrored pair fails? I always thought a mirrored zfs pool will be used for boot drive so one failed everything would go as normal, no?

 

[joeschmuck]

What you are creating is a Software RAID not a hardware RAID and this is a big difference. Let me expain a little more. When you setup your computer you likely had to go into the BIOS to tell it which device to boot from. This device was your SATADOM. Lets say you add a second SATADOM, well how do you tell the BIOS to boot from it as well, or if the first one fails. Lets say you select the second SATADOM as the second device to boot from. That sounds good however that would mean that the first SATADOM would have to be removed from the system or the system must think it's removed. So now you have a failure of your first SATADOM and it's in the form of data corruption, you system started to bootstrap from it but then it hit bad data and your system hangs. Lets say your first SATADOM just looks blank, well you BIOS will likely still wait around not doing anything.

Now lets say your second SATADOM fails, well your system woudl still boot but FreeNAS will complain to you (hopefully) that you have a SATADOM failing, it may not be obvious which one either.

If you had a true hardware RAID card and created a Mirror then there is proper hardware on that card to handle any drive failure and boot right on though it. I've done it and found that it wasn't worth the extra heat produced by this extra card. It's just too simple to maintain a backup of your config file and reinstall FreeNAS to a new device and then upon the first bootstrap, restore your config file and all will be good in the world.

My advice is, if it ain't broke, don't fix it. Just make a backup of your config file just in case you need it. And again, there is a copy in you pool but it does take a few manual step to pull it out.

I hope that helps.

 

[Ender117]

What you are creating is a Software RAID not a hardware RAID and this is a big difference. Let me expain a little more. When you setup your computer you likely had to go into the BIOS to tell it which device to boot from. This device was your SATADOM. Lets say you add a second SATADOM, well how do you tell the BIOS to boot from it as well, or if the first one fails. Lets say you select the second SATADOM as the second device to boot from. That sounds good however that would mean that the first SATADOM would have to be removed from the system or the system must think it's removed. So now you have a failure of your first SATADOM and it's in the form of data corruption, you system started to bootstrap from it but then it hit bad data and your system hangs. Lets say your first SATADOM just looks blank, well you BIOS will likely still wait around not doing anything.

Now lets say your second SATADOM fails, well your system woudl still boot but FreeNAS will complain to you (hopefully) that you have a SATADOM failing, it may not be obvious which one either.

If you had a true hardware RAID card and created a Mirror then there is proper hardware on that card to handle any drive failure and boot right on though it. I've done it and found that it wasn't worth the extra heat produced by this extra card. It's just too simple to maintain a backup of your config file and reinstall FreeNAS to a new device and then upon the first bootstrap, restore your config file and all will be good in the world.

My advice is, if it ain't broke, don't fix it. Just make a backup of your config file just in case you need it. And again, there is a copy in you pool but it does take a few manual step to pull it out.

I hope that helps.

I see. you were talking about rebooting to replace the drives. I got hot swappable bays so I can just change it on the fly (I am not OP just passer by

 

[pro lamer]

you were talking about rebooting to replace the drives.

Well, let me in... boot drives are usually used during booting :) Unless the op has the system dataset there... Do you @rmccullough ?

Sent from my mobile phone

 

[joeschmuck]

I see. you were talking about rebooting to replace the drives. I got hot swappable bays so I can just change it on the fly (I am not OP just passer by

What does this have to do with mirrored boot devices? I hope you don't think you can hot swap the boot device, not something I'd ever recommend. Also you should not hot swap a data drive unless it is mission critical to keep the system up. Not all hot swap drive bays are created equal, even some of the good ones can still fail. Trayless drive bays are risky.

 

[Ender117]

What does this have to do with mirrored boot devices? I hope you don't think you can hot swap the boot device, not something I'd ever recommend. Also you should not hot swap a data drive unless it is mission critical to keep the system up. Not all hot swap drive bays are created equal, even some of the good ones can still fail. Trayless drive bays are risky.

What's the problem? If boot device is mirrored, replacing a failed member drive should not break anything. There are slim risks that 2nd one fails before reslivering complete, but then it's just a boot drive anyway.
Also would like to hear why you consider hot swapping drives risky. IMHO the biggest risk is you pulled the wrong drive, as long as you have proper hardware.

 

[Stux]

Freenas bootdrives contain multiple partitions. One of them is a zfs partition and contain the majority of the OS files. This is mirrored.

The other contains the bootstrap which has enough on it to find and mouth the zfs boot partition. AFAIK this is not mirrored. It’s just an identical copy across all members.

And then there are the boot sectors.

Interestingly booting freenas off one boot disk and having it mount a freenas boot pool from a different disk is certainly possibly.

So, short story long, mirrored boot drives mean you have two bootable disks to choose from with two bootstraps and then a mirrored zfs os in the bios.

If the currently selected boot device fails enough you’ll get a failed boot rather than a failure to find a boot device and perhaps fallback to the other one. But you’ll be able to select the other one. Then that will be able to boot and find the zfs boot pool, and possibly even the one on the failed device, but it won’t rebuild the bootstrap.

Removing and reading a boot mirror device will rebuild the bootstrap (by copying).

Also, freenas will scour boot devices for configuration when reinstalling into them... which is neat.

 

[rmccullough]

Well, let me in... boot drives are usually used during booting :) Unless the op has the system dataset there... Do you @rmccullough ?

Sent from my mobile phone

I honestly don't know. I am fairly new to FreeNAS, pools, data sets, etc. Not trolling, I just don't know. If it helps, here is what my boot environments looks like:

And my boot status:

Based on the smiley you could be asking me rhetorically, but I just don't know.

My motherboard (Supermicro X9DRi-LN4F+) manual says it supports RAID 0, 1, 5, 10 for the onboard SATA ports. The drives which make up my FreeNAS Pool are connected via SAS backplane to a pair of LSI 9210-8i controllers, not the motherboard. My SATADOM is currently connected to one of the 2 SATA 3.0 ports.

The point of my original post was to try and build some more redundancy into my home FreeNAS setup. My understanding is that SATADOM's are prone to failure due to their single-chip design. The tradeoff is small form factor and speed without needing a drive bay/mount, but the downside is higher failure rate than an SSD.

I would really like to avoid "modifying" my case to mount an additional SSD (or pair of SSD's). It is a nice case, Supermicro CSE-826A-R1200LPB.

So what I am hearing is that creating a bootable USB Stick (motherboard only supports USB 2.0 btw) would be a very cost effective method of creating a backup boot disk should my SATADOM fail. The intent would be that it provides a temporary solution while I restore my boot disk redundancy by either:
a) get another SATADOM purchased and setup
b) set up a 2nd bootable USB stick

What I was hoping to hear is whether the additional cost of a 2nd SATADOM (~$55 for SATADOM vs ~$10-15 for a USB Stick) was worth it. I assume it would be faster, but only at boot or upgrade? Would it be better since the motherboard apparently supports hardware RAID. Does this mean it would seamlessly switch between boot devices if the primary fails? How would I be notified if one fails?

 

[Ender117]

Freenas bootdrives contain multiple partitions. One of them is a zfs partition and contain the majority of the OS files. This is mirrored.

The other contains the bootstrap which has enough on it to find and mouth the zfs boot partition. AFAIK this is not mirrored. It’s just an identical copy across all members.

And then there are the boot sectors.

Interestingly booting freenas off one boot disk and having it mount a freenas boot pool from a different disk is certainly possibly.

So, short story long, mirrored boot drives mean you have two bootable disks to choose from with two bootstraps and then a mirrored zfs os in the bios.

If the currently selected boot device fails enough you’ll get a failed boot rather than a failure to find a boot device and perhaps fallback to the other one. But you’ll be able to select the other one. Then that will be able to boot and find the zfs boot pool, and possibly even the one on the failed device, but it won’t rebuild the bootstrap.

Removing and reading a boot mirror device will rebuild the bootstrap (by copying).

Also, freenas will scour boot devices for configuration when reinstalling into them... which is neat.

I am assuming you are replying to me, so...

Code:

root@freenas:~ # gpart show
=>	  40  83886000  da0  GPT  (40G)
		40	  1024	1  freebsd-boot  (512K)
	  1064  83884968	2  freebsd-zfs  (40G)
  83886032		 8	   - free -  (4.0K)

I guess the "freebsd-boot" is what you called bootstrap and "freebsd-zfs" is the OS partition.
Now correct me if I am wrong, my understanding is that after POST, "freebsd-boot" is loaded into RAM, which enables the actual ZFS partition for OS to be loaded and FreeNAS start to boot.

Now, on an up and running FreeNAS, why would offline and replacing a member of the mirrored boot device cause a problem? To me this can be either:
1.For some reason the bootstrap used in boot needs to be read/wrote/accessed while system is already running, and if you yank out the wrong drive (or that one fails) then yeah the system would panic.
2.The OS partition is not loaded as a pool containing mirror vdev. It loads "freebsd-zfs" from one drive as "master" and syncs everything to the "slave". So if the master becomes unavailable (failed drive, pulled out by error, etc.), the system will panic.
Both seems to be pretty strange design, and in either case even with a mirrored boot device a single failed drive may bring down the whole system. This pretty much deficit the idea of mirroring your boot device. Maybe I am missing something?

 

[Ender117]

I honestly don't know. I am fairly new to FreeNAS, pools, data sets, etc. Not trolling, I just don't know. If it helps, here is what my boot environments looks like:
View attachment 25845

And my boot status:
View attachment 25846

Based on the smiley you could be asking me rhetorically, but I just don't know.

My motherboard (Supermicro X9DRi-LN4F+) manual says it supports RAID 0, 1, 5, 10 for the onboard SATA ports. The drives which make up my FreeNAS Pool are connected via SAS backplane to a pair of LSI 9210-8i controllers, not the motherboard. My SATADOM is currently connected to one of the 2 SATA 3.0 ports.

The point of my original post was to try and build some more redundancy into my home FreeNAS setup. My understanding is that SATADOM's are prone to failure due to their single-chip design. The tradeoff is small form factor and speed without needing a drive bay/mount, but the downside is higher failure rate than an SSD.

I would really like to avoid "modifying" my case to mount an additional SSD (or pair of SSD's). It is a nice case, Supermicro CSE-826A-R1200LPB.

So what I am hearing is that creating a bootable USB Stick (motherboard only supports USB 2.0 btw) would be a very cost effective method of creating a backup boot disk should my SATADOM fail. The intent would be that it provides a temporary solution while I restore my boot disk redundancy by either:
a) get another SATADOM purchased and setup
b) set up a 2nd bootable USB stick

What I was hoping to hear is whether the additional cost of a 2nd SATADOM (~$55 for SATADOM vs ~$10-15 for a USB Stick) was worth it. I assume it would be faster, but only at boot or upgrade? Would it be better since the motherboard apparently supports hardware RAID. Does this mean it would seamlessly switch between boot devices if the primary fails? How would I be notified if one fails?

Hi OP, sorry that I may have hijack your post. The system dataset is usually in the main pool, or at least recommend to be there. You can search the manual on how to change the setting.

Now on adding a 2.5 SSD to your system, you can just tape them to somewhere in your chassis, they generate very little heat and vibration. They are usually much more reliable than USB sticks and cheaper than sata DOMs.

How important is the uptime of your FreeNAS box? If you can stand a 30min downtime you can just keep a backup of your config. When your boot drive fails, use a USB stick as your temporary boot device while you wait for replacement.

 

[joeschmuck]

Also would like to hear why you consider hot swapping drives risky. IMHO the biggest risk is you pulled the wrong drive, as long as you have proper hardware.

Lets say you have a trayless design which is quite popular. While installing the drive the SATA Power and SATA Data connectors do not line up perfectly and end up shorint out the system and POP goes the power supply, burns the hard drive SATA connections, and of course the drive tower. I've witnessed this first hand on a Supermicro tower but it was a tray design and there was a tiny amount of side to side slop built into this design, well it didn't work perfectly. Yes, you buy a name brand and it is still possible to screw it up. This was a very expensive repair, well it wasn't my money thankfully. So this is one way a hot swap could go bad and probably the worst way and it the perfect reason to power down if at all possible. So I will always recommend the safest way possible, but if someone needs to hot swap, well just take your time.

Now on adding a 2.5 SSD to your system, you can just tape them to somewhere in your chassis, they generate very little heat and vibration. They are usually much more reliable than USB sticks and cheaper than sata DOMs.

Maybe some good velcro but not just tape. The heat in the case can cause normal tape to loose it's adhesion, well Gorilla Tape might do very well, that is some serious stuff!

The system dataset is usually in the main pool, or at least recommend to be there.

This is the default setup when you create your first pool.

And again, my advice is to use a SSD as your boot device but since you have a perfectly good SATADOM right now, just conmtinue to use that for now. As @Ender117 pointed out, if you do end up with a failure, just stick in a USB flash drive, install FreeNAS to it, restore your configuration file, order a new boot device, maybe the SSD this time. You may not need to deal with this failure for several years.

 

[rmccullough]

Sounds good. I will look into a way to regularly backup my configuration. I know I could LMGTFY, but where is the config file located?

 

[Ender117]

Lets say you have a trayless design which is quite popular. While installing the drive the SATA Power and SATA Data connectors do not line up perfectly and end up shorint out the system and POP goes the power supply, burns the hard drive SATA connections, and of course the drive tower. I've witnessed this first hand on a Supermicro tower but it was a tray design and there was a tiny amount of side to side slop built into this design, well it didn't work perfectly. Yes, you buy a name brand and it is still possible to screw it up. This was a very expensive repair, well it wasn't my money thankfully. So this is one way a hot swap could go bad and probably the worst way and it the perfect reason to power down if at all possible. So I will always recommend the safest way possible, but if someone needs to hot swap, well just take your time.

I see, I will be using FreeNAS for VM storage so shutting it down means almost everything had to go down with it. I will take the risk of hot swap. Is there any other reason one should not replace a member of the mirrored boot drive?

 

[pro lamer]

There are some edge cases when system dataset cannot be moved off the boot pool. For example: all other pools encrypted. If your dataset has to stay on the boot drive then you may want to avoid USB drives (since the system dataset is subject to lots of writes).

And me neither can recall from the top of my head how to check where the system dataset is located. But yeah, it's in the manual...

I can also remember that there is some additional setting letting/making extra reports stored to the system dataset thus causing more data being written to the system dataset...

Sent from my mobile phone

 

[pro lamer]

Removing and reading a boot mirror device will rebuild the bootstrap (by copying).

Will you elaborate or link more on this please? What and when would read and rebuild?

Sent from my mobile phone

 

[Stux]

Will you elaborate or link more on this please? What and when would read and rebuild?

Sent from my mobile phone

re-adding