Help with Boot Drive Setup Choices

[NASbox]

I am planning to upgrade my FreeNAS Box (FreeNAS-8.3.1-RELEASE-x64 (r13452) ) to the current Version 11 and I need some guidance as to the best alternative for Boot Volume.

I am attempting to choose between 2 Mirrored USB Flash Drive and a Single SSD.

In 8.3 I am using a single v8.3 USB Flash drive which houses all the boot code and all the configuration information. I very much like the idea of keeping the system code/config separate from data pools because it eases maintenance and backup. This model is also very easy to clone, and I have an hot swap ready to go should the drive fail (which it did once after about 2 years of service). Given the cheapest quality SSD I can easily buy locally is 120GB, I would like to get some speed / operational benefits from the SSD.

Questions:
How much write traffic does the Boot volume see (assuming that it doesn't have the database on it)?
If I use a single SSD is it OK to keep the database on the boot volume?
If I use a single SSD will it improve GUI response speed (and/or system performance)?
If I use a single SSD can the system be tuned to use the SSD for swap?
If I use a single SSD is it easy to clone the last snapshot and configuration to a flash drive for quick/easy restore to a new SSD if the SSD happens to fail?

As I understand things so far the characteristics of these two choices are:
(Anything I am missing?)

2 Mirrored USB Flash Drive
Cons:
- Destroyed by high write volume

Pros:
- Cheaper
- Doesn't use a SATA Port

Single SSD
Pros:
- Much more robust / able to tolerate a reasonable volume of write traffic
- Musch faster than USB drives (don't know if this is meaningful other than faster boot time)

Cons:
- More Expensive
- Wastes a SATA Port
- No redundancy

Thanks in advance for any assistance/input/suggestions.

 

[nightshade00013]

An SSD doesn't need the same redundancy a flash drive does. Also the flash drives are written to more now so they will not last near as long as they used to.

As far as the SSD going bad it will honestly probably last a lifetime as the FreeNAS boot drive but you can always use a config backup script to backup the settings to the pool. Then just reinstall and upload the config and you are in business. https://forums.freenas.org/index.php?threads/backup-config-file-every-night-automatically.8237/

All things said while I am using USB drives right now if I make any changes to my system or build out another one I will be using a small SSD as the boot drive. Not only is it faster booting but updates install faster. Right now updating with USB drives is like starting up an old DOS era computer. Hit the update button and come back in 30 to 40 minutes and see if it's ready to go or not.

 

[joeschmuck]

How much write traffic does the Boot volume see (assuming that it doesn't have the database on it)?

Not a terrible amount. But more than 8.3.1.

If I use a single SSD is it OK to keep the database on the boot volume?

I assume you are talking about the configuration file. Of course it would be on the boot device.

If I use a single SSD will it improve GUI response speed (and/or system performance)?

Not GUI performance but it will improve system speed with respect to software updates. Also system performance in respect to system reliability since SSDs rarely die as compared to USB Flash drives.

If I use a single SSD can the system be tuned to use the SSD for swap?

Could or Should is the question. Yes it could but No you shouldn't. If your system is properly sized for the RAM then the SWAP file should never be required, and a few GB of hard drive space is considered acceptable should you really need the SWAP space.

If I use a single SSD is it easy to clone the last snapshot and configuration to a flash drive for quick/easy restore to a new SSD if the SSD happens to fail?

All you need to do is follow the common practice to backup your configuration file periodically when you make changes to FreeNAS. If you need to recover from a boot device failure all you need to do is install FreeNAS and after rebooting you would restore your config file. It's really that simple.

Cons:
- More Expensive

Yes but not terribly expensive at all. The cheapest one you can buy will do, just need 16GB or more. And old SSD will do very well.

- Wastes a SATA Port

It's only a waste if you need it for a data drive.

- No redundancy

Don't need it. It's not like a mirrored FreeNAS boot device would boot if the primary drive failed, odds are they system would not boot and just hang there. The only way around something like this is a true RAID card mirror which is designed to operate during a true failure of a drive.

For me it's a no-brainer, go with a SSD if you have the SATA port. you won't regret it unless you get a SSD with an infant mortality issue, we are all subject to this type of problem.

 

[danb35]

All you need to do is follow the common practice to backup your configuration file periodically when you make changes to FreeNAS.

...which is automatically done anyway onto the .system dataset, which by default is moved to the data pool as soon as one is created.

 

[NASbox]

Thanks all for the input...

As far as the SSD going bad it will honestly probably last a lifetime as the FreeNAS boot drive but you can always use a config backup script to backup the settings to the pool. Then just reinstall and upload the config and you are in business. https://forums.freenas.org/index.php?threads/backup-config-file-every-night-automatically.8237/

Thanks that's very helpful.... I will definately do this.

If I use a single SSD can the system be tuned to use the SSD for swap?

Could or Should is the question. Yes it could but No you shouldn't. If your system is properly sized for the RAM then the SWAP file should never be required, and a few GB of hard drive space is considered acceptable should you really need the SWAP space.

I agree swap shouldn't happen very often, but if it did happen, would far rather have it happen on an SSD at 400-500MB/s, than spinning disks at 100 MB/s.

How does choice of swap happen? Does it go to one drive, or does it get scattered around to many different pools/drives/volumes?

For me it's a no-brainer, go with a SSD if you have the SATA port. you won't regret it unless you get a SSD with an infant mortality issue, we are all subject to this type of problem.

I was thinking that--cost me 65+tax ~ $74CDN for 120GB SSD... at this time that was the cheapest I could find at this time. Major overkill, but if I can make logs a bit more persistent it would be nice.

...which is automatically done anyway onto the .system dataset, which by default is moved to the data pool as soon as one is created.

What happens if the data pool is exported/disconnected (for maintenance)?

Is there any reason not to move the .system dataset to the boot drive?
(I did a quick test on a VM, and if I recall correctly it is possible to move the .system dataset to boot drive.)
Is it OK to do this? If not, why not?

I would prefer to keep system stuff separate from data and have the system able to run from the SSD.

 

[gpsguy]

One of the primary reasons for the swap file, is to allow for disk replacements when the replacement disk isn't exactly the same size as the original. If the replacement is a few byte too small, FreeNAS can adjust the size of the swap file to accommodate the new drive.

I agree swap shouldn't happen very often, but if it did happen, would far rather have it happen on an SSD at 400-500MB/s, than spinning disks at 100 MB/s.

 

[NASbox]

One of the primary reasons for the swap file, is to allow for disk replacements when the replacement disk isn't exactly the same size as the original. If the replacement is a few byte too small, FreeNAS can adjust the size of the swap file to accommodate the new drive.

Thanks for the comment. I understand that, and I don't mind having the space sit idle for that reason. It's just if the system actually needed to swap, I'd rather swap to an SSD than a spinning disk which was the reason for my question.

 

[nightshade00013]

The other thing you have to realise is that the pool will likely be faster than an SSD. My RaidZ3 with 7 drives kicks the crap out of the SSD I have in my desktop. The place where the pool lags behind is IOPS but in all honesty it's not going to matter much when the swap file is rarely used anyway.

https://calomel.org/zfs_raid_speed_capacity.html

The other thing is keeping it simple, while you CAN technically build a computer using transistor tubes it's so complex it's not easy to deal with. The more that you tweak and futz with things that work out of the box the more likely you are to have issues later on. And when you do have those issues the people here trying to help are going to request a reversion of those changes to simplify things and help ferret out the problem. It's kinda like the issue we have where and average home user wants to get fancy and add SLOG and L2ARC to their pool but only have 8GB of ram among other things then come and complain when the system doesn't meet their expectations.

You are more than free to do what you want but you have had someone say it's not a great idea so you can either listen or go it on your own and deal with what comes after.

 

[joeschmuck]

How does choice of swap happen?

This is not something a person can do via the GUI, this is all CLI and smoke and mirrors. And I agree that the SSD would typically not be as fast as the swap file spread across the entire pool.

Does it go to one drive, or does it get scattered around to many different pools/drives/volumes?

When you are asking these type of questions, which is fine honestly, it means to me that you do not understand enough of the underlying system to figure out how to make this work. For a non-standard setup like this I personnaly feel the user needs to do a ton of research and figure it out, not be handed a set of instructions and be left to your own destruction.

I was thinking that--cost me 65+tax ~ $74CDN for 120GB SSD... at this time that was the cheapest I could find at this time. Major overkill, but if I can make logs a bit more persistent it would be nice.

I agree, the 120GB that you are looking at cost too much. I was looking on Amazon for Canada and found several 32GB SSDs for under $30CDN. I'm sure Ebay has used ones as well. These cheaper models use older controller chips so they are not as speedy as the new stuff on the market but they kick the crap out of a USB Flash drive still and should be more reliable.

(I did a quick test on a VM, and if I recall correctly it is possible to move the .system dataset to boot drive.)

This is in the User Guide, you should read it from cover to cover a few times, lots of great information in there.

I am planning to upgrade my FreeNAS Box (FreeNAS-8.3.1-RELEASE-x64 (r13452) ) to the current Version 11

What are your system specs? Since you are running a fairly old version of FreeNAS, it may be prudent to disclose this data so we can tell you if there are any pitfalls before you make the change.

One of the primary reasons for the swap file, is to allow for disk replacements when the replacement disk isn't exactly the same size as the original. If the replacement is a few byte too small, FreeNAS can adjust the size of the swap file to accommodate the new drive.

Glad you brought that one up, I forgot about that key thing. A few people have been bitten by this before when they changed the swap space to a zero value.

 

[NASbox]

The other thing you have to realise is that the pool will likely be faster than an SSD. My RaidZ3 with 7 drives kicks the crap out of the SSD I have in my desktop. The place where the pool lags behind is IOPS but in all honesty it's not going to matter much when the swap file is rarely used anyway.

https://calomel.org/zfs_raid_speed_capacity.html

Interesting read .... I'm curious why the read speed on a 4 drive raidz2 is slower than a single disk:

Code:

1x 4TB, single drive,		   3.7 TB,  w=108MB/s , rw=50MB/s  , r=204MB/s
4x 4TB, raidz2 (raid6),		 7.5 TB,  w=204MB/s , rw=54MB/s  , r=183MB/s 

I'm actually running a 4 disk raidz2. When I move to a bigger box, I'll likely go to 6 disks when I get some spare cash.

The more that you tweak and futz with things that work out of the box the more likely you are to have issues later on. And when you do have those issues the people here trying to help are going to request a reversion of those changes to simplify things and help ferret out the problem. It's kinda like the issue we have where and average home user wants to get fancy and add SLOG and L2ARC to their pool but only have 8GB of ram among other things then come and complain when the system doesn't meet their expectations.
You are more than free to do what you want but you have had someone say it's not a great idea so you can either listen or go it on your own and deal with what comes after.

Thanks, that's why I'm posting here... is to get this type of feedback from those with experience. Some things are easy/relatively safe to do, and others are not worth the effort/risk. I'll defer to your judgement on this and not mess with the swap.

Does this same comment apply to .dataset? I believe it's easy/possible to move .dataset to the boot drive in the GUI. (I made a small test system on VirtualBox just to see what V11 looked like and if it would upgrade from v8 like the manual says it does. I was using virtualized ZFS disks, and I was able to put the .dataset on the boot drive and it seemed to work ok - but this was just a quick test, not production.)

If I use an SSD boot is there any good reason not to put the .dataset on it?

[Update]
Hi joeschmuck

I appreciate the lengthy response. Based on the feedback I'm going to leave the swap alone.

This is in the User Guide, you should read it from cover to cover a few times, lots of great information in there.

I assume you are referring to section 5.6 System Dataset - I saw the caution regarding putting the dataset on USB flash drives, but I didn't see anything about not putting the dataset on a SSD boot drive. Did I miss something?

I agree, the 120GB that you are looking at cost too much. I was looking on Amazon for Canada and found several 32GB SSDs for under $30CDN. I'm sure Ebay has used ones as well. These cheaper models use older controller chips so they are not as speedy as the new stuff on the market but they kick the crap out of a USB Flash drive still and should be more reliable.

I didn't see anything name brand... maybe I missed it. But for $20/30 (Add shipping and the difference is even less) I wasn't going to risk some cheap piece of junk from China that could trash my system and cost me a few hours time to restore at best and data loss at worst. I just bit the bullet and bought the 120GB drive. I'll likely keep several updates and a lot more log data than I would otherwise. Bigger drives tend to be faster for some reason, and will likely wear better as well because there are lots of free cells to rotate through.

What are your system specs? Since you are running a fairly old version of FreeNAS, it may be prudent to disclose this data so we can tell you if there are any pitfalls before you make the change.

System specs are as per my profile except I'm going to drop an i7-2600 (picked up used from Kijijii and another 16GB of ram that I have laying around - I hope to eventually run a jail/VM or two.

FreeNAS-8.3.1-RELEASE-x64 (r13452)
ASUS P8Z68-M Pro - Pentium G620, 16GB DDR3 ( https://www.asus.com/Motherboards/P8Z68M_PRO/ )
RAIDZ2: WD60EFRX x 4
RAIDZ HDS724040ALE640 x 2

 

[Stux]

If you have an m.2 slot, another option is to use a small/cheap m2 drive. The intel optane m2 drives are relatively affordable.

If you want to put swap on your boot disk, the simplest option is to use a swap file instead of a swap disk.

The real issue with swap on your data disks is that if a disk fails, and swap is in use (even a little bit) then your entire system may crash.

I have a script to try to help mitigate this risk

Also, I documented how to move swap to a dedicated device here

The same technique could be used to dynamically disable swap and create/enable the file based swap.

 

[Stux]

If you want to put swap on your boot disk, the simplest option is to use a swap file instead of a swap disk

Oh look... someone documented it...

Resource: How to relocate swap to the boot drive

 

[NASbox]

Thanks for the input Stux... I think I'll err on the side of caution and just leave the swap alone. Given I'm no BSD Ninja, I don't want to take the chance of messing things up. I'll be happy with just getting the dataset on the SSD.