How best to use 960GB Optane in FreeNAS build?

[victorhooi]

I am putting together a new FreeNAS system, built around the SuperMicro A2SDi-H-TP4F board. The reason I got this board was for the onboard 10G (SFP+) connectivity.

It currently has 4 x 16GB of DDR4 ECC RAM, although I may possibly increase this to 4 x 64GB (if that's the advice of people here).

I have 8 x 12TB HDDs in there.

I have a single M.2 HP EX950 2TB drive I was going to use as a boot drive, and possibly to store ZVOLs for Bhyve? (Not sure best way to achieve this though).

I also have an Intel Optane 905P 960GB drive I was hoping to use as a standalone ZIL and SLOG device.

My question is - what is the best way to use the 960GB Optane drive? If I read this page it seems to suggest 960GB is too large - however, this is the drive we have, so would like to know the best way to use it.

Also, for the M.2 drive - what is the best way of using this? (I assume it's likewise too large for a boot drive, so don't want to waste the rest. Am I better off using a USB 3.0 drive as a boot disk?)

 

[Chris Moore]

I have a single M.2 HP EX950 2TB drive I was going to use as a boot drive

There is no excuse for using a 2 TERABYTE drive as a boot drive when FreeNAS only needs 32GB.
That is pure insanity.

I also have an Intel Optane 905P 960GB drive I was hoping to use as a standalone ZIL and SLOG device.

You may not need either of those drives, depends on what you plan to use this NAS for. What is the use case?

What kind of chassis are you using?

By what means are you connecting the data drives to the system board?

Am I better off using a USB 3.0 drive as a boot disk?)

No. For reliability reasons, you should never boot from USB.

A mirrored pair of mechanical drives like this would be a good solution if you have a space to mount them:
https://www.ebay.com/itm/Hitachi-Travelstar-40GB-Internal-5400-RPM-2-5-HTS541640J9SA00-SATA-Hard-Drive/264244921757

 

[victorhooi]

There is no excuse for using a 2 TERABYTE drive as a boot drive when FreeNAS only needs 32GB.
That is pure insanity.

Yes, I figured it was too much storage, which is why I was thinking of using part of it for Bhyve VMs maybe?

(The board has a single M.2 slot, and I figured this was the best use for that since USB boot drives can be finicky).

You may not need either of those drives, depends on what you plan to use this NAS for. What is the use case?

What kind of chassis are you using?

The use case is a bit open ended but it would include things like media serving, document/photograph archiving, possibly photo/video editing (e.g. Lightroom or Premiere), and may be also using this FreeNAS for VMs (assuming Bhyve is ready).

Might also use it for archiving VM disk images, and forensic disk images.

I did consider simply using the machine as Proxmox with a FreeNAS, but I believe the general consensus on the forums is that this is a bad idea?

The case is a uNAS 810.

By what means are you connecting the data drives to the system board?

The eight drives are connected by mini-SAS to SATA cables, directly to the motherboard. (This SuperMicro motherboard has four SATA ports and two miniSAS ports which each break out to four SATA cables)

No. For reliability reasons, you should never boot from USB.

I could possibly shove a cheap 2.5" SSD on there for this. There is room in the chassis and we do have SATA ports available.

Is there some way of splitting up the Optane and M.2 drives to get more usage from them, and increase performance?

 

[Chris Moore]

Yes, I figured it was too much storage, which is why I was thinking of using part of it for Bhyve VMs maybe?

There is no option to partition the boot drive and use part of it for something else. If you want to use this drive for something besides booting, just get something else to boot from and separate the function physically, not by partitioning.

Is there some way of splitting up the Optane and M.2 drives to get more usage from them, and increase performance?

In what way are you planning to share data to the network? If you are going to use sync write, it might be good to have the Optane for SLOG. The 2TB SSD could be used for storing your VMs. It depends on the fine details of the configuration.

 

[victorhooi]

The network share will be done using SMB, for Windows and MacOS clients.

I believe SMB is async by default right?

Is there still any advantage to using the Optane as a SLOG? Or what other uses are there? Any other thoughts for this setup?

Does L2ARC.make sense? (Currently 64GB RAM but I am looking at going to 4x64GB if required to make use of the Optane)

 

[Chris Moore]

I believe SMB is async by default right?

SMB is just the communication protocol and although it does default to async, because it is faster, it can handle sync write requests. Here is a link:
https://www.systutorials.com/241743/is-samba-sync-or-async-for-writes/
For Windows, it occasionally has a sync request but most are async. That is not true for Mac, as I understand it, Mac OS does a lot of sync write requests. I don't normally deal with Mac clients, but I was reading some messages on the forum recently.

Is there still any advantage to using the Optane as a SLOG?

This is a bit of an involved answer. The thing I did on my NAS was to manually partition a NVMe SSD to have a swap partition, and a partition for SLOG and a partition for L2ARC, then I set the default swap space to zero, so no swap partition is created on any of my storage pool disks, then I created my pool so there is no swap space in the pool. This saves me from the trouble of dealing with swap partitions when replacing a failed hard drive in the storage pool. FreeNAS automatically recognizes and uses the swap partition on the NVMe drive at boot and I manually added the SLOG and L2ARC partitions to the pool. This configuration has been working well for me, almost a year now, but it has many steps involved in the configuration that are far outside the default configuration of a FreeNAS system.
I can give you directions for the correct way to partition the Optane drive if this is something you are interested in trying.
For my boot pool, I use a mirrored pair of mechanical drives because speed and size really don't help anything for the boot pool.
I have a different brand but this is basically what I am using for booting my system:
https://www.ebay.com/itm/Hitachi-Tr...-HTS541640J9SA00-SATA-Hard-Drive/264244921757

 

[victorhooi]

Hi Chris,

That sounds like an interesting setup, and possibly what I want to do. I assume you found some kind of performance improvements from this setup? =)

I would have to delete my existing pool, though, right, to remove the swap space?

I'm keen to do that - are you able to provide any steps to help me partition my Optane NVMe drive (960GB), and set it up to the existing pool?

Thanks,
Victor

 

[Chris Moore]

I assume you found some kind of performance improvements from this setup?

The big advantage that I got was not having to deal with swap space on my data disks. That had been a recurring problem every time there was a disk failure. It is less a problem now that some changes have been made in how swap space is utilized, but I find it best to not have swap space on the data disks.

I would have to delete my existing pool, though, right, to remove the swap space?

It is the easy way to delete the swap space, but not the only way. Have you already loaded data into your pool?

I'm keen to do that - are you able to provide any steps to help me partition my Optane NVMe drive (960GB), and set it up to the existing pool?

I am not sure what device name your Optane drive has, I was using an Intel SSDPEDMD400G4 PCIe NVMe drive in my system, and the device might be named slightly different. Here are the commands I used to partition the drive:

Code:

gpart create -s gpt nvd0
gpart add -i 1 -b 128 -t freebsd-swap -s 16g nvd0
gpart add -i 2 -t freebsd-zfs -s 20g nvd0
gpart add -i 3 -t freebsd-zfs nvd0

Then I grep for the glabel:

Code:

glabel status | grep nvd0

My pool name is Emily, so I use that and the gptid to add the SLOG and L2ARC to the pool:

Code:

zpool add Emily log gptid/ae487c50-bec3-11e8-b1c8-0cc47a9cd5a4

zpool add Emily cache gptid/ae52d59d-bec3-11e8-b1c8-0cc47a9cd5a4

Here is a link to the post where I talked about it and posted some graphics showing my test results:
https://www.ixsystems.com/community...s-partitioned-for-two-pools.62787/post-483761

 

[Mastakilla]

I recently managed to buy a second hand Intel Optane 900p 280GB for a nice price. So I'm now also one of those silly persons who has an Optane drive without really needing it and looking to give it some purpose ;)

SLOG
Although I did see a slight decrease in performance vs no sync / no slog, I am considering configuring "sync" to "always" and using it as a SLOG, to get that little extra "data-in-transit-safety".
I'm planning to use SMB (dataset) for my main share (to Windows desktops) and iSCSI (zvol) for a OneDrive share. The VMs running on my desktop, I'll try to keep stored on the local-desktop-ssd (as long as space permits). If I can get bhyve to work a bit better (I'm still having trouble with my non-US keyboard atm), I might also start running some VMs on FreeNAS.
So yes, I realize that I probably don't really need a SLOG (yet?), but unless someone strongly advises against, I'll probably configure it anyway.

L2ARC
As I only have 32GB RAM and quite a large pool, I will not configure it as L2ARC atm. I probably need my RAM more as ARC atm...
I also have some pricewatches active for more RAM, so perhaps someday I'll obtain 64GB and then I'll reconsider...

SWAP
I did find this idea very interesting, so I found below resource from @Stux :

How to relocate swap to an SSD or other partition

FreeNAS by default will allocate 2GiB on each pool HD and use this as swap. Unfortunately, if a disk fails, your system may crash. If you have an SSD, or another device/partition for use as swap it can be beneficial to relocate swap to that...

www.ixsystems.com

However, the command no longer seems to work, as the mirrored swap on the pool isn't stored in /etc/fstab anymore and also "swapoff -a" no longer seems to work on it:

How to relocate swap to the boot drive

I noticed the release notes for 11.1 showed 'Support for mirrored swap partitions'. Does this occur automatically (and is there way to verify/determine) or is this something that has to be configured through the gui/cli? I did not find a reference in the manual, and I'm curious what 'support'...

www.ixsystems.com

Also, as the "pool-swap" is now mirrored, one of the most important reasons to do this (a potential FreeNAS crash in case of a HDD failure) no longer is there.
However with my system, I might have a bit of swap usage / performance benefit from it though, as I only have 32GB RAM for a 80TB pool. But I didn't really confirm / test this yet...

@Chris Moore
A "Wizened Sage", as yourself, apparently still does this, do you still see potential benefit in this? Can you explain how you set this up?
Also you say "It is less a problem now that some changes have been made in how swap space is utilized, but I find it best to not have swap space on the data disks."
Do you mean you have set "System - Advanced - Swap size in GB" to 0? Or are you using some trick like @Stux suggests in his resource (using a post-init script)?

FreeNAS VM Boot Device
I've just also figured out that it is possible to create a pool from a single partition on the Optane disk, using the "zpool create" command, then exporting it and importing it in the GUI.

I can then use it for storing some less-important VMs (yes, I am aware there is no redundancy).

I did notice that the GUI isn't really "meant" for this. Because when I choose "Export / Disconnect" this pool and select "Destroy data on this pool", it not only "clears" the partition used for the pool, but it removes ALL partitions on the Optane disk (so also the above mentioned swap and SLOG partition).

Except for being aware of the above "risks", is there any other reason not to do this?