Replacing Synology

[MikeyG]

Would love some feedback on the build I'm looking at.

I currently have a Synology DS2015xs backing up via rsync to another FreeNAS box I built pretty cheaply with slow drives and no ECC just to learn FreeNAS and give myself a backup machine. That Synology currently has 8 ST2000NM0033 2TB drives in single RAID5 equivalent. Goal is to get rid of it and move to something with more performance (seems to be CPU constrained sometimes), better redundancy, and something that doesn't make annoying vibrating sounds in my closet with the bays full (crappy case design maybe)?

I've currently got the Synology, FreeNAS backup box, and my workstation running 10gbe off a Netgear switch. Most of my current storage is large movies, and I really like being able to move them to my workstation and work with them very rapidly if I want to. I'm currently able to get about 700-800gbps there off the Synology via SMB and would like to maintain that level of performance.

I also occasionally use iSCSI on the Synology for VMs or other misc applications, but I wouldn't consider iSCSI performance super important to me - just a consideration that I don't want it to suck. It would be nice to be able to run some VMs from VMware off the FreeNAS box, but those likely won't be high demand.

What I'm looking at:

Motherboard: Supermicro X11SSH-TF
CPU: Core i3-7100
PSU: Seasonic 650W Gold
Memory: 1X Crucial 16GB DDR4 2400 CT16G4WFD824A (Intent to upgrade later once prices come down.)
Case: iStarUSA D-400-6 with 8 bays hot swap

Was going to boot this off two sandisk ultrafit usb drives in mirror.

Aside from just storage, I will be running a few jails/plugins. I don't need to run VMs on it, or Plex right now, although it would be nice if I could. I can always upgrade the CPU later, but I assume a dual core i3 at 3.9GHZ is enough for now?

One of my questions is around storage config:

It would seem that RaidZ2 is my best bet for bandwidth performance and redundancy. However, I will definitely be needing to purchase larger drives this year, and I don't want to purchase 8 of them all at the same time. I really like the idea of doing 4 paired mirrors, and buying them 2 at a time to upgrade over say the next 2 years. The only other way I see to upgrade in chunks would be 2X 4 drive vdevs in RaidZ2 but that seems like a waste of storage and I worry about performance. I could do 2X RaidZ1, but when my drives end up being over 8TB each in the future, having to do a rebuild or in place upgrade will stress me out a bit with only single disk redundancy per vdev. I understand that mirrors would give me the best of IOPS (which I don't really need but is always nice), but how would they affect bandwidth for large files compared to one large RaidZ2?

The motherboard has 8 SATA ports which I think would be perfect, but if I wanted to boot from SSDs instead of USB or leave room for a SLOG or L2ARC (neither of which I think I'll use) I'm out of SATA ports and the case technically has space for 12 3.5" drives total. I'm thinking I could always get a PCIe SSD if it comes down to it or another SATA adapter. There is also the X11SSH-CTF with another 8 SAS ports, but for more money and I'm not sure how complex it will be to get the two storage controllers to cooperate together.

Only other consideration for me is that the motherboard and CPU are getting a little old. Coffee Lake supporting motherboards from Supermicro might be out later this year, and I could get a quad core i3 for the same price as the Kaby Lake. But, I'm not convinced extra CPU power is worth the wait for me, and not sure what features those new motherboards could bring that would make any difference.

Thoughts?

 

[jgreco]

Mirrors have better bandwidth, but two-way mirrors have less redundancy than RAIDZ2. If you wish to retain redundancy even in the event of a failure you do have the option to move to three-way mirrors. This offers an awesome amount of read capacity (because each disk can be doing separate read tasks) and IOPS. The read bandwidth and IOPS for mirrors is generally better than that of RAIDZ. But I can also pretty much guarantee most users don't need that capacity. You seem to have a handle on the other dimensions of that, so I'll let it be with that comment.

You don't really need a board any newer than an X9. Intel CPU performance has been relatively stagnant over the last six to eight years. Since Sandy Bridge, the Intel CPU's have mostly been slight evolutionary changes between versions, with the notable exception of the E3 bump to 64GB and the whole Spectre/Meltdown thing. A Sandy Bridge E3-1230 CPU benchmarks at 7907 while the new E3-1230v6 comes in at 10212 so you might be going "that seven year old part is only 77% as fast" but this disregards that the speed jumped from 3.2 to 3.5GHz, so in a head-to-head at the same clock speed, that old CPU is 85% of today's CPU. It's not obsolete. It's just well tested. Do not get super-OCD over it unless there's actually a feature you need. So feel free to buy an X11 with confidence, it isn't going to be significantly bested by a newer CPU tomorrow unless Intel does something radical like cut their pricing and double their cores. But if someone comes along with a killer deal on an X10 or even an X9, *do it* and know that you're never likely to be able to tell the difference.

 

[MikeyG]

Thanks for your response jgreco!

Sounds like I don't need to worry about the CPU/motherboard from a "new tech" perspective which makes sense.

It also sounds like going with mirror pairs only has upside when it comes to performance, but no real downside? Are there situations where RAIDZ2 will perform better than mirrors give the same number of drives?

Does it seem fair to think about a pool made of two-way mirrors as having a level of redundancy in between RAIDZ1 and RAIDZ2, since I can lose more than 1 drive, (maybe up to 4 in my case), but not any two drives like RAIDZ2? If that is the case, I think I'm comfortable with that level of risk given that I have backups, and a worst case scenario where there is downtime to restore doesn't really bother me. I'd rather prioritize the added performance and my ability to upgrade drives more easily in the future.

 

[jgreco]

We've reached that interesting technological point where people have been talking about Moore's Law breaking down, AND it actually appears like that is matched by a general plateau in performance. We need to increase core count and look at better efforts to parallelize, and that's showing in the higher end CPU's. But that's not really important for NAS unless you're moving craptons of traffic. The general sizing of the CPU is probably important for things like Plex, but the difference is between "I've got two slow 2GHz cores" and "I've got two fastish 3.4GHz cores" and "I've got four 3.4GHz cores", not "this CPU is 14% slower". It's the big increments that are relevant.

ZFS was designed around mirrors. RAIDZ was a later addition. The reason for RAIDZ was that mirrors are hella-expensive especially if you go up to three-way mirrors, and many people do not want or need the extreme performance, they just want what's called "near-line" storage so that they can store and retrieve things, quickly, but also only occasionally, and most importantly, CHEAPLY.

I'm not coming up with any realistic cases where RAIDZ has a performance advantage. I guess if you were write-flooding the system, you would be pushing more data to mirrors than RAIDZ and could probably contrive a way to demonstrate "RAIDZ can be faster" on that basis (PCIe bus full, 24Gbps wideport full, etc). There's probably other edge cases like that.

It's also worth understanding that RAIDZ uses variable space to store parity which leads to threads like this one: https://forums.freenas.org/index.php?threads/iscsi-using-twice-the-space.61123/#post-434657

But if you're contemplating VM storage, the best advice I have for you is to go with mirrors, get drives big enough that you have GOBS of free space on your pool, and you will also have the flexibility to do smaller upgrades in the future.