New Pool Design

[jasonsansone]

I wanted to solicit opinions on pool design for a new NAS configuration.

Fixed Specs:
Chassis: SuperMicro CSE-864
Motherboard: SuperMicro X9DRi-F
CPU: 2x E5-2697 v2
RAM: 16 x 16GB 1033mhz ECC (256GB total)
NIC: NC560SFP+ for 10Gbe and Intel i350 for 1Gbe
Hard Drives: 18x Shucked WD white label 5400* rpm 10TB
Compression: zstd (waiting on RC1 to implement)
Record Size: 1Mb

I have one Sun (Intel) P3605 1.6TB drive for L2ARC and a mirror of two P3605 1.6TB drives for metadata and small blocks. All blocks 64Kb and under will be on the P3605's. I have measured the amounts, it will easily fit. I have two Windows VM's in bhyve and several jails (Plex, Sonarr, Radarr, Lidarr, NZBGet, qBittorrent, Tautulli, Bazarr, YouTube-dl, Jackett). Jails and VM's were previously on a separate pool of mirrored SSDs, but I want to move them to the main pool.

I can either go with a pool of mirrors or three vdevs of six drives in RAIDZ2. RAIDZ2 will result in the IOPS equal to 3 drives with 120TB usable space. A pool of mirrors will yield IOPS equal to 9 drives and 90TB of usable space. I plan to store ~70TB of data (at present) on this NAS. It is backed by ZFS replication to a second NAS which is comprised of two vdevs of six 14TB drives in RAIDZ2. I am not concerned about the differences in parity between a pool of mirrors vs striped vdevs of RAIDZ2 since I have a complete, online backup.

I am primarily concerned about IOPS. I re-encode all media to x265 (load spread across four VM's on four physical CPU's) and tend to remux a lot of discs. Prior to TrueNAS 12, I would have easily chosen mirrors. Here is the curve ball - Fusion Pools. Since I have a mirror of P3605's to store all small blocks and metadata, how much difference will I see with RAIDZ2 vs Mirrors? Has anyone tested or played with the performance changes from implementing Fusion pools? Can I safely assume that a pool of mirrors comprised of 18x drives and a P3605 for metadata and small blocks will saturate 10GBe?


* Might be a lie.

 

[Samuel Tai]

All things being equal, a pool of mirrors should usually beat a RAIDZx pool for IOPS. The problem in your case is the fill line. For 70 TB, you'll fill your mirror pool to about 78%, but your RAIDZ2 pool to 52%, and given the fill rate, they should actually both run about the same for IOPS. Given the mirror pool would be close to the 80% limit, you'll be better off with the RAIDZ2 pool, and have 30% headroom left over.