Boot drives and l2arc/zil question

[jbeez]

Looking to update my existing server a little bit. Currently has 31gb usable ram, 5 4tb drives in a raidz1, and booting off of two thumb drives.

I was thinking of adding normal ssd drives for the boot drives and maybe storing some of my plugin jails on them or something else instead of the slow disk?

My options were some 500gb samsung evo drives, or getting a pci express card to install MVNe drives on, or sata doms.

Didn't know if the pciexpress card route would even work in freenas, also not sure whats preferable over the other. Any recommendations for boot drives and cache drives?

Thanks,

 

[sretalla]

Boot is all-encompassing for a disk in FreeNAS. You can't use the boot disk(s) for anything else (except swap with some trickery well documented by stux).

Jails on SSD is a great idea (and works well).

Plenty of folks have PCIe cards for fast storage as well as connecting SSDs to SAS/SATA HBA PCIe cards.

I have also used PCIe cards to adapt additional M2 NVME sticks... although one of those has needed replacing after 9 months of medium use (the adapter card, not the stick).

 

[Chris Moore]

I was thinking of adding normal ssd drives for the boot drives

That is a fine idea, but don't plan to use the boot drives for anything other than booting. In fact, there is not much point in using SSDs for it because there is no real advantage in the speed over a mechanical disk. I have been using a pair of mechanical hard drives in both of my home NAS systems for almost three years now with zero problems. Something like this should work nicely:
https://www.ebay.com/itm/Hitachi-Tr...-HTS541640J9SA00-SATA-Hard-Drive/264244921757

 

[Chris Moore]

Didn't know if the pciexpress card route would even work in freenas

It does, with the right hardware, but there is no point in doing that for the boot drive. You might use it for your jails / VMs though.

Take a look at this suggestion from @jgreco :
https://www.ixsystems.com/community...donics-ad3m2spx4-m-2-to-pcie-converter.39735/

 

[jbeez]

Thanks for the replies, looking into these today.

If freenas just chews up the entire boot disk, is there really any reason than for me to move it off of the usb thumb drives I'm using now?

Also I know one of the cache drives is restricted to 16gb, but I also read it might not be useful under 32gb of ram, I seem to be borderline there.

 

[Chris Moore]

If freenas just chews up the entire boot disk, is there really any reason than for me to move it off of the usb thumb drives I'm using now?

Reliability. USB memory sticks can't be tested or checked for their SMART drive status so there is no way to predict when they might fail. They just randomly die at the most inopportune moments. Mechanical drives can be tested like any other mechanical drive and will last for a long time, and when they do need to be replaced, you will likely have some errors to tell you before it just suddenly quits working.

 

[2nd-in-charge]

Any recommendations for boot drives and cache drives?

I currently use mirrored low end 120Gb SSDs as boot devices, but in the future I'd probably be getting used Intel 80Gb S3500 SSD for the same purpose, which are available on ebay for about the same price as new low end SSDs.

Also I know one of the cache drives is restricted to 16gb, but I also read it might not be useful under 32gb of ram, I seem to be borderline there.

That's not quite right. Cache is not restricted to 16Gb, neither is SLOG.
There is L2ARC (which is a cache) and there is SLOG (which is a sync writes log).
Slog is the one that isn't going to use more than 16G (won't even use that), but larger drives are faster and have better endurance.
SLOG must have power loss protection, most SSDs are not suitable.
You probably don't need a SLOG, but it you decide to get one, find a used Intel s3700 SSD. NVME drives make much better SLOG devices due to low latency, but good ones are quite expensive, even used.
L2ARC can be any consumer SSD with decent read speed. The problem that you read about is that L2ARC requires an index which reduces amount of RAM available as ARC.
Check your ARC stats in Reporting->ZFS. If your ARC size is close to your memory size, and cache hits are low, L2ARC might help. Otherwise don't bother with that either.
The rule of thumb for its size is x5-x10 your Ram, so about 240Gb. You can get any NVME drive, really, but a new Samsung 970 Evo or a used 950 Pro won't break the bank and will be plenty fast.

What's the rest of your server like? Are there many SATA ports available?

 

[Chris Moore]

You probably don't need a SLOG, but it you decide to get one, find a used Intel s3700 SSD. NVME drives make much better SLOG devices due to low latency, but good ones are quite expensive, even used.

This is the model I use in my server:
https://www.ebay.com/itm/Intel-DC-P...-MLC-HHHL-AIC-20nm-SSDPEDMD400G4/303136710113
I have partitioned it with three partitions, one for swap, one for SLOG and one for L2ARC. It works well enough for me. You can use the utility from Intel to over-provision it if you want.
@Stux made a post on that: https://www.ixsystems.com/community...n4f-esxi-freenas-aio.57116/page-4#post-403374

 

[FN-Boba]

Reliability. USB memory sticks can't be tested or checked for their SMART drive status so there is no way to predict when they might fail. They just randomly die at the most inopportune moments. Mechanical drives can be tested like any other mechanical drive and will last for a long time, and when they do need to be replaced, you will likely have some errors to tell you before it just suddenly quits working.

if a thumb drive or hdd boot drive crashes, what kind of effects would we expect for the rest of the system? would plugging in a fresh drive with FreeNas on it get you back up and running or is there more to it? i'm talking strictly file access

 

[Chris Moore]

if a thumb drive or hdd boot drive crashes, what kind of effects would we expect for the rest of the system?

In my personal experience with using USB boot media, which I stopped doing because of this, a failure of either of the USB boot modules, even in a mirrored pair configuration, would cause the system to lockup and crash. The USB interface is simply less robust than SATA and does not deal with the device failure in a graceful manner.

would plugging in a fresh drive with FreeNas on it get you back up and running or is there more to it?

Not on USB. I switched to using SATA drives for boot media and I did some testing with my system by pulling the plug on a drive while the system was running. I could not test for every eventuality, but I was able to remove a SATA drive from the boot pool while the system was running, install a replacement drive and resilver the boot pool and the system remained up and operational through the entire operation. That was using a server grade system board with SATA ports that support hot-swap.

i'm talking strictly file access

If you are talking about storage, a failure to the boot pool will crash the server. There is no way that I could find to avoid a crash with USB boot drives. That is the reason I switched from using USB boot media to using SATA boot media.

EDIT: For clarity, I am discussing a "boot pool" made up of more than one drive. FreeNAS is NOT able to operate without a functioning boot pool.

 

[FN-Boba]

Not on USB. I switched to using SATA drives for boot media and I did some testing with my system by pulling the plug on a drive while the system was running. I could not test for every eventuality, but I was able to remove a SATA drive from the boot pool while the system was running, install a replacement drive and resilver the boot pool and the system remained up and operational through the entire operation. That was using a server grade system board with SATA ports that support hot-swap.

that is kind of remarkable, no operational boot drive and the system still functioned? upon boot does FreeNas run completely in RAM then??

 

[Constantin]

I believe Chris was playing with a boot pool consisting of multiple SATA disks for which he intentionally disabled one of the drives to see what would happen. As I understand it, USB sticks do not react as gracefully to this type of scenario as SATA disks.

 

[Chris Moore]

that is kind of remarkable, no operational boot drive and the system still functioned?

With a mirrored pair of drives, ONE of the two drives was removed, as a test.

upon boot does FreeNas run completely in RAM then??

No. There was a time, long ago, when it did but it does not do that now.

To be absolutely clear, the boot pool (a group of more than one disk) was still online and available. I was able to remove ONE and only one disk from a mirrored PAIR of (two) disks. This works on SATA, but when I tried the similar experiment using USB media, the system crashed within seconds.

 

[jbeez]

Thanks for all the replies, I haven't abandon this, I'm troubleshooting some issues! I'm leaning towards picking up a used r710 atm

 

[jbeez]

Ok so, I have an r710 coming tomorrow, it’ll be outfitted with 6 3.5” drive bays, a single ssd where the cdrom goes, 120gb ecc ram, and I’m either going to move my existing drives and install over, or do a fresh install on new 8TB drives and take it from there. I need to see what kind of slot I’ll have available on the box. This box is going to limit my available options for boot drives. So I need to look into the pci-e card for the stick ssd drives

 

[2nd-in-charge]

do a fresh install on new 8TB drives and take it from there.

Probably this. Will give you an opportunity to move to RaidZ2 from RaidZ1 you've been running.

This box is going to limit my available options for boot drives. So I need to look into the pci-e card for the stick ssd drives

There should be two SATA ports in R710. You can replace ODD with an SSD caddy. That would be one boot drive. For the other you can get one of those dual M-key and B-key PCIe cards. The B-key could be the SATA M.2 drive (your boot mirror), and the M-key your NVME ssd (cache or slog if you wish).

Edit: Or leave the ODD alone and get this:
https://www.ixsystems.com/community...donics-ad3m2spx4-m-2-to-pcie-converter.39735/
You'll need to disconnect the ODD from its SATA port though..

 

[jbeez]

This is really moving in another direction. Here's what I have currently and I have decisions to make based on space available.

Server: dell r710 2*xeon X5670 6core ea, 120GB ram, h200 card flashed to IT mode, although its not working in the dedicated storage slot, Id like to figure that out because I'm already running out of slots.

NIC: I have an Intel 350 nic card in there, I could take that out but isn't the igb driver much better and this a better card than the onboard broadcom with the bce driver?

HDs: I have a pci-e adapter card now with a samsung 500gb 970 evo plus nvme m.2, a transcend 64gb m2 for the other slot in this card. pny 120gb ssd in an adapter that goes where the optical is. The transcend and the pny will be my new boot devices maybe, unless I need this space for something else....
6X8TB seagate HD's for a big raidz2 pool to store media on.

So far this is 3 out of 4 pci-e slots used. I could also add an external sas card and add a drive cage to fit more storage. I have one sitting here not in use, just need the card and cables.

So I don't know if I need jails/plex transcode on an ssd, or if setting up cache drives for my zpool would effectively handle that concern for me? Basically what is the best use of my extra storage outside of the spinning disks, I have limited room. I could still use USB sticks to boot if I wanted, and I do have a samsung 500gb ssd I could put in that optical slot instead of the 120gb pny. I could also pick up one of those dc 3700 pci-e cards. Do I need to worry about redundancy with any of these extra drives(besides the boot drives, I know I want redundancy there).

Server basically is there for plex/BT to serve up to friends and family and myself. And I may throw a few jails on there for other non-essential things. It's just for home. Might NFS share some directories, may add a vmware server.


Another question, my server has 24gb of pc3 10600(1333mhz) and 96gb of pc3 8500(1066mhz). Running them in mixed mode the server just runs it at 800mhz. Is it worth keeping that 24gb in there if it drops the speed from 1066 to 800? Related, I could pickup another 96gb of the 10600 if it would help with the speed of jails/transcodes/file access by caching alot more.

 

[2nd-in-charge]

Server: dell r710 2*xeon X5670 6core ea, 120GB ram, h200 card flashed to IT mode, although its not working in the dedicated storage slot, Id like to figure that out because I'm already running out of slots.

https://www.ixsystems.com/community...ard-found-in-the-internal-storage-slot.74149/

NIC: I have an Intel 350 nic card in there, I could take that out but isn't the igb driver much better and this a better card than the onboard broadcom with the bce driver?

FWIW we've been running FreeNAS on Dell T710 with on-board Broadcom NIC for over two years now without any issues.

I could still use USB sticks to boot if I wanted,

SSDs are more reliable.

So I don't know if I need jails/plex transcode on an ssd, or if setting up cache drives for my zpool would effectively handle that concern for me?

Why is it a concern? Transcode is CPU intensive. Your HDD array should have more than enough speed to serve movies.

I could also pick up one of those dc 3700 pci-e cards.

If you want to get one of those, make sure you get a good price. They are still good, but Optane drives are faster, and prices for new 280GB 900p Optane drives are comparable to some dc P3700. But then for a home server you might be quite happy with a cheap 16Gb or 32Gb Optane drive.

Do I need to worry about redundancy with any of these extra drives

I think you'd find that most people running expensive PCI-e SLOG devices are only running one of them. I've got my DC S3700 SATA drives in mirrored slog, but they were cheap enough.
The L2ARC cache drive doesn't need redundancy.

Might NFS share some directories, may add a vmware server.

That's when a Slog could become useful.

my server has 24gb of pc3 10600(1333mhz) and 96gb of pc3 8500(1066mhz). Running them in mixed mode the server just runs it at 800mhz. Is it worth keeping that 24gb in there if it drops the speed from 1066 to 800? Related, I could pickup another 96gb of the 10600 if it would help with the speed of jails/transcodes/file access by caching a lot more.

I believe in FreeNAS world more RAM is better than faster RAM. I don't think 800MHz restriction is due to the mismatch in memory speeds. It's because in those servers using the third bank of RAM limits the memory speed to 800MHz:
https://www.dell.com/downloads/glob...ver-pedge-installing-upgrading-memory-11g.pdf
I think you have plenty of RAM for the home server. Set it up and watch your ARC stats. You can always add more RAM and/or L2ARC drive later.

 

[jbeez]

I’m not super concerned about the transcodes, this is already better than what I’ve been using. Im just trying to make sure I get the essentials right on the initial build and make sure I understand.

I read that about the ram, but I’m not using the third slot yet. It shipped with 6 4gb sticks. I added 6 16gb sticks. So I still have 6 open ram slots. I got that 96gb ram for about $165 figured it was a good deal considering the density.

Thanks for sharing the vid I’m going to watch, the issue I’ve been having seems to be information overload. I’ve found several different guides and they’re all a little different along with the versions of the tools so I’ve been getting mixed results.

 

[jbeez]

Got my card working........... the storage slot was disabled in the bios! I flashed that card more times than I care to admit... finally flashed the dell firmware and put it into the slot and it still wasn't showing up, checked the bios and it was disabled.