Necroing my thread a bit, but I'm finally narrowing down on a real build and I thought I'd reach out once again for some advice. I'm still not sure if I'll be going with FN10 on ESXi, or just FN10 native with bhyve to handle the limited VMs I need, so I'm keeping things open to support both. (read: planning to VT-d everything, even if I end up not needing to).
I've given up on the Skylake platform entirely. I just wasn't happy with the fact that my expandability now and forever would be "a slightly faster quad-core processor". For that reason I've settled on the LGA-2011 platform, in fact I'm hoping there's a nice E5-2650v3 headed my way from China right now! I should never need more than 10c/20t, but if I'm wrong, there's always a chance to find a 22c/44t monster CPU for ~$100 in 5+ years :D
I'm still going back and forth on boards, which is where I'm looking for some advice. Whatever board I buy I intend to install three PCIe boards now, and I'd like there to be an extra slot or two for expansion down the road:
- LSI 9210-8i - I acquired one from another FreeNAS user, pre-flashed to IT mode P20, with two 8087 cables and shipping included for less than the going rate of an M1015 on Ebay. VT-d to FN, obviously.
- Mellanox Connectx2 10GbE - I know the history on these cards is terrible, but unless I'm mistaken, support was finally added earlier this year, FN9.10, in May or so? I plan on running point to point w/ a DAC between my primary and backup servers, just to speed up backups/restores and keep that traffic off the main LAN. If it doesn't work, I resell it or it's ~$30 down the drain. I've wasted a lot more than that experimenting with PCs...
- Intel 2xGbE NIC - Hoping for a cheap i350 that's not an obvious counterfeit. The plan is to VT-d the onboard i210's to FN and let the i350 handle ESXi and the other VMs.
My two top options on boards right now are the X10SRL-F and X10DRL-i. Both have dual i210 NICs, 8x RDIMM, 10x SATA, etc. etc. etc.
The SRL is a single socket board, and a big draw is the PCIe expandability. Four x8 slots (two in x16) plus two x4 slots in x8's that you can either use as they are or steal the lanes to make one a fifth x8 slot and leave the other with no lanes. There's also a PCIe 2.0 x4 in a x8, which would be fine for adding the GbE NIC. I can't find anything not to love about this board..?
The DRL is very similar, the primary difference being the second CPU socket, and the PCIe complement is one true x16, three x8's, and two x4's in a x8 (one of them PCIe 2.0 - for the NIC again). The price difference is about $60, $275 vs. $335, and I don't think for my workload there's any value in spending $60 for the option of adding a 2nd CPU down the road? Not to mention, running with only one CPU in this board you lose access to four of the RAM slots and one of the x4 PCIe.
If I up my budget about $100 over the X10SRL, there are two other single-socket options, the X10SRH-CF and X10SRH-CLN4F.
The former has an onboard SAS3008 and upgrades the i210's to i350-AM2. It also has six PCIe slots, but overall you lose one x8 and one x4 compared to the SRL, and pick up a PCIe 2.0 x2 in a x4 instead. I like the NIC upgrade and the SAS3 could be nice if I hadn't already bought a 9210. Of course I could probably resell the 9210 to cover most of the cost of the board upgrade, but I'm not sure it's worth the trouble since my expander backplane is only SAS2...
The -CLN4F version is identical to the -CF, but with an i350-AM4 for two extra LAN ports. If they could be split up and VT-d separately that'd be great and save me from buying an add-on NIC, but since they can't it's overall a worthless upgrade. Scratching that one off the list.
As for upgraded dual-socket options, they're either way too expensive or too big for my chassis. The only reasonable upgradeis the X10DRi, and it's ~$400 ($70 more than DRL-i), with nothing but an i350-AM2 upgrade and more memory slots I'll never use.
If anyone has any thoughts or experiences with these boards let me know. I've searched through the forums and read a few build threads, in particular ones with the X10SRL-F since it's my front-runner right now. Unfortunately, this platform isn't nearly as popular as the "regular" X10's with i3's and such, since it's total overkill if all you're looking for is a NAS. ;)