Variations of this topic get posted from time to time, but I didn't find my exact question. Sorry for the long post.
tldr;
1) Intel 905P Optane in only a mirror configuration vs triple mirror as a metadata only Fusion Drive.
2) A single mirror vdev is easy, but I could probably fit a few more as striped mirrors or a pair of triple mirrors. Thoughts on configuration and reliability are appreciated.
I'm considering grabbing some 960GB/U.2 Intel 905P Optanes for a Fusion Drive or L2ARC (metadata only?). These are on sale atm and while not the datacenter grade, I believe they will be sufficient for my use case, unless there's a solid argument against them. My primary interest is making huge directory searches, listings, thumbnails, etc. much faster than the HDD pool can support. I believe one of these two are the best option beyond RAM (which is maxed out on RDIMMs for me and I don't want to pay the premium for used 8x 256GB DIMMS).
I have a server chassis with 45 drives (3 rows of 15 drives, 2x ZRAID2 of 7 drives plus 1 spare per row), EPYC w/512GB RAM, 2x 25GbE bonded. VMs are on a separate stripped mirror pool of SSDs. I have three PCIe slots left and a pair of 4x Oculink internal ports. My metadata runs typically <0.5% of the on zpool data. Almost everything is async, so no SLOG. I also have no L2ARC right now, but have considered it. I have a pair of 2.5" drive slots left, but beyond that, I have to get creative (which is fine if the value is there).
The easy thing to do is to put a pair of 960GB Optanes as a mirror and move my metadata only onto them (ZFS send since I have room on the HDD array to hold two copies temporarily). Conceptually, with a metadata use rate of <0.5%, that would support approximately 200TB (my current capacity). However, this would only be a mirror and it's often recommended to use a triple mirror for metadata. In theory these drives are so much more reliable than HDDs I should be well protected. The typical argument against a mirror is that once one drive fails, you don't have two copies to compare and so any errors propagate forward. However, given the reliability of Optane and ZFSs robust checksums, I assume a mirror is a reasonable risk for my pool configuration?
Alternatively, I could put these as a striped L2ARC (not pool critical) or I could get a bit creative with adding more internal U.2 drive slots, a Broadcom 9600 HBA, and several more 905Ps to get a triple mirror; at that point, it's probably worth going with four or six mirrored drives.
tldr;
1) Intel 905P Optane in only a mirror configuration vs triple mirror as a metadata only Fusion Drive.
2) A single mirror vdev is easy, but I could probably fit a few more as striped mirrors or a pair of triple mirrors. Thoughts on configuration and reliability are appreciated.
I'm considering grabbing some 960GB/U.2 Intel 905P Optanes for a Fusion Drive or L2ARC (metadata only?). These are on sale atm and while not the datacenter grade, I believe they will be sufficient for my use case, unless there's a solid argument against them. My primary interest is making huge directory searches, listings, thumbnails, etc. much faster than the HDD pool can support. I believe one of these two are the best option beyond RAM (which is maxed out on RDIMMs for me and I don't want to pay the premium for used 8x 256GB DIMMS).
I have a server chassis with 45 drives (3 rows of 15 drives, 2x ZRAID2 of 7 drives plus 1 spare per row), EPYC w/512GB RAM, 2x 25GbE bonded. VMs are on a separate stripped mirror pool of SSDs. I have three PCIe slots left and a pair of 4x Oculink internal ports. My metadata runs typically <0.5% of the on zpool data. Almost everything is async, so no SLOG. I also have no L2ARC right now, but have considered it. I have a pair of 2.5" drive slots left, but beyond that, I have to get creative (which is fine if the value is there).
The easy thing to do is to put a pair of 960GB Optanes as a mirror and move my metadata only onto them (ZFS send since I have room on the HDD array to hold two copies temporarily). Conceptually, with a metadata use rate of <0.5%, that would support approximately 200TB (my current capacity). However, this would only be a mirror and it's often recommended to use a triple mirror for metadata. In theory these drives are so much more reliable than HDDs I should be well protected. The typical argument against a mirror is that once one drive fails, you don't have two copies to compare and so any errors propagate forward. However, given the reliability of Optane and ZFSs robust checksums, I assume a mirror is a reasonable risk for my pool configuration?
Alternatively, I could put these as a striped L2ARC (not pool critical) or I could get a bit creative with adding more internal U.2 drive slots, a Broadcom 9600 HBA, and several more 905Ps to get a triple mirror; at that point, it's probably worth going with four or six mirrored drives.