Sanity Checking new Build

[Chris Moore]

I used similar logic when selecting an e5 x10 platform.

The e5-1620v4 is approximately equivalent to an e3-1230v5. I wouldn't use the 2609. It's s poor cpu. Low speed low core count.

The next step up is the 1650v4, which is a great 6 core CPU which runs up to 4ghz (iirc).

And then, maybe in a few years you can find an eBay special with 20 cores ;)

I wouldn't bother with a Dual Processor board.

I know it is just an opinion and all that goes with that, but why not just buy the lowest cost option that will get the job done and in a few years (with the money you didn't waste now) you can buy another low cost option that will get the job done then. I see the people where I work spending money every budget cycle buying hardware that they think will be "future proof" and it never works out. They just bought 200 systems at a cost of about $10000 each that will still be obsolete in 5 years and all they did was dump more money on the fire.

Just look at the workload you actually have right now, because you will certainly predict the future workload inaccurately and the technology will change.

 

[Stux]

Because my workload involves Samba. And samba is single threaded, and thus 1.8ghz would be a bottleneck.

The 1650 handles today's workload, and what I expect to be the workload for the future.

If we need more cores down the road because we're doing more virtualization, then there is still that option

 

[Magius]

Necroing my thread a bit, but I'm finally narrowing down on a real build and I thought I'd reach out once again for some advice. I'm still not sure if I'll be going with FN10 on ESXi, or just FN10 native with bhyve to handle the limited VMs I need, so I'm keeping things open to support both. (read: planning to VT-d everything, even if I end up not needing to).

I've given up on the Skylake platform entirely. I just wasn't happy with the fact that my expandability now and forever would be "a slightly faster quad-core processor". For that reason I've settled on the LGA-2011 platform, in fact I'm hoping there's a nice E5-2650v3 headed my way from China right now! I should never need more than 10c/20t, but if I'm wrong, there's always a chance to find a 22c/44t monster CPU for ~$100 in 5+ years :D

I'm still going back and forth on boards, which is where I'm looking for some advice. Whatever board I buy I intend to install three PCIe boards now, and I'd like there to be an extra slot or two for expansion down the road:

1. LSI 9210-8i - I acquired one from another FreeNAS user, pre-flashed to IT mode P20, with two 8087 cables and shipping included for less than the going rate of an M1015 on Ebay. VT-d to FN, obviously.
   
2. Mellanox Connectx2 10GbE - I know the history on these cards is terrible, but unless I'm mistaken, support was finally added earlier this year, FN9.10, in May or so? I plan on running point to point w/ a DAC between my primary and backup servers, just to speed up backups/restores and keep that traffic off the main LAN. If it doesn't work, I resell it or it's ~$30 down the drain. I've wasted a lot more than that experimenting with PCs...
   
3. Intel 2xGbE NIC - Hoping for a cheap i350 that's not an obvious counterfeit. The plan is to VT-d the onboard i210's to FN and let the i350 handle ESXi and the other VMs.

My two top options on boards right now are the X10SRL-F and X10DRL-i. Both have dual i210 NICs, 8x RDIMM, 10x SATA, etc. etc. etc.
The SRL is a single socket board, and a big draw is the PCIe expandability. Four x8 slots (two in x16) plus two x4 slots in x8's that you can either use as they are or steal the lanes to make one a fifth x8 slot and leave the other with no lanes. There's also a PCIe 2.0 x4 in a x8, which would be fine for adding the GbE NIC. I can't find anything not to love about this board..?
The DRL is very similar, the primary difference being the second CPU socket, and the PCIe complement is one true x16, three x8's, and two x4's in a x8 (one of them PCIe 2.0 - for the NIC again). The price difference is about $60, $275 vs. $335, and I don't think for my workload there's any value in spending $60 for the option of adding a 2nd CPU down the road? Not to mention, running with only one CPU in this board you lose access to four of the RAM slots and one of the x4 PCIe.

If I up my budget about $100 over the X10SRL, there are two other single-socket options, the X10SRH-CF and X10SRH-CLN4F.
The former has an onboard SAS3008 and upgrades the i210's to i350-AM2. It also has six PCIe slots, but overall you lose one x8 and one x4 compared to the SRL, and pick up a PCIe 2.0 x2 in a x4 instead. I like the NIC upgrade and the SAS3 could be nice if I hadn't already bought a 9210. Of course I could probably resell the 9210 to cover most of the cost of the board upgrade, but I'm not sure it's worth the trouble since my expander backplane is only SAS2...
The -CLN4F version is identical to the -CF, but with an i350-AM4 for two extra LAN ports. If they could be split up and VT-d separately that'd be great and save me from buying an add-on NIC, but since they can't it's overall a worthless upgrade. Scratching that one off the list.

As for upgraded dual-socket options, they're either way too expensive or too big for my chassis. The only reasonable upgradeis the X10DRi, and it's ~$400 ($70 more than DRL-i), with nothing but an i350-AM2 upgrade and more memory slots I'll never use.

If anyone has any thoughts or experiences with these boards let me know. I've searched through the forums and read a few build threads, in particular ones with the X10SRL-F since it's my front-runner right now. Unfortunately, this platform isn't nearly as popular as the "regular" X10's with i3's and such, since it's total overkill if all you're looking for is a NAS. ;)

 

[Chris Moore]

I've given up on the Skylake platform entirely. I just wasn't happy with the fact that my expandability now and forever would be "a slightly faster quad-core processor". For that reason I've settled on the LGA-2011 platform, in fact I'm hoping there's a nice E5-2650v3 headed my way from China right now! I should never need more than 10c/20t, but if I'm wrong, there's always a chance to find a 22c/44t monster CPU for ~$100 in 5+ years :D

Edit... Sorry. I can't see why you need all those cores. I would go with a higher clock speed lower core count CPU even if I picked the socket 2011 board.

I don't know what you plan to do with this, but to me it is massive over kill, like if I put a v10 in my Honda Civic.

I share 12 TB of drive space to my LAN, and run 3 virtual machines with my Emily NAS. I have had no performance issues with trans-coding videos in PLEX, even while running VMs using VirtualBox. The only thing I wish I had was a little more memory, but I have had no problems that I would attribute to a lack of memory.

You said that you have already made your choice on CPU (on the way from China), so any advice here is too late but I would have gone for a higher clocked processor as many tasks are single threaded. The extra cores may not provide much help for you. My CPU is only quad but with 8 threads it has all the horsepower I need. You would have to run a lot of VMs to need more because they are not likely to be fully engaged at all times.

As and example, I am running 30% CPU utilization right now because I am doing an RSYNC between the two NAS units AND at the same time, doing a disk wipe one of the drives in preparation for removing it. Normally, it is less than 10%...

 

[Magius]

I don't know what you plan to do with this, but to me it is massive over kill, like if I put a v10 in my Honda Civic.

Hehe, I read your pre-edit response and realized that I left out a very important piece of information last night. That's what I get for posting around midnight I guess. :D

Keeping to a budget on this build is a high priority, and yes, spending $900-$1200 on a CPU would be asinine. I don't blame you for not pulling punches on that. What I neglected to mention is that the Xeon on the way from China is a late-QS model with essentially retail stepping. I got it for $150, around half the price of the cheapest decent OEM Xeon I could find, which would have been a 4c/8t 1620 (low cores high clock, as you recommend). I saw some reports on another forum of people using this particular stepping QS chip successfully, so I decided to take a chance. I really don't like those 1600 series as they draw way too much power for what they are, and while GHz is important for NAS, I believe the 2650 will do fine with a little turbo. If it doesn't pan out the way I expect, I'll resell it and settle for the 1620, no real harm done.

Also high priority to me is future upgradability. Hence why the Skylake platform felt so limiting in terms of CPU, Memory and PCIe expansion. My current server is 8 years old and still more than meets my NAS needs. What is no longer meeting my needs is my 12-year old backup server, plus I have new needs for a VM host, so my plan is retire the current server to backup duty and build a new NAS/VM server. That should keep me going for another 10 years hopefully, as long as I do it right. All that said, while the 10c/20t is supreme overkill for 'just NAS' it was too tempting to pass on given my current needs for a few VMs, and my desire for LONG term usability of this new server.

Hopefully that clarifies what must have seemed a completely irrational decision, lol. And yes, I understand the risks of the ES/QS chips. I plan to beat on it quite a bit during the build phase to build up some faith that it works correctly, plus keep my important data in 3-4 places, which is just good practice anyway...

In any case thank you for your response, which is certainly 'correct', I just weighed my options and decided to go a different direction. :)

 

[Stux]

I'd skip the dual processor boards.

Two processors are slower than a processor which is twice as fast.

The memory and PCI buses have to hop from one CPU to the other if the code is running on the far core.

So if you don't need crazy ram or 44 cores, then you probably don't need a dual processor.

X10-SRL sounds pretty good.

 

[Stux]

I know it is just an opinion and all that goes with that, but why not just buy the lowest cost option that will get the job done and in a few years (with the money you didn't waste now) you can buy another low cost option that will get the job done then

Because my time, building and commissioning the server, is worth more than the money 'saved' by under speccing and building two or three times.

 

[Magius]

I'd skip the dual processor boards.
X10-SRL sounds pretty good.

Thanks Stux. I didn't quote your whole post but I agree with it completely and it's exactly where I'm leaning. I don't see any real advantage to spending $60-$100+ on a second CPU socket, just for the possibility of going to a crazy number of cores in the future. The CPU I found has way more cores than I'll have a use for anytime soon, even if I experiment with some kind of VDI at home, which is not yet on the radar. On top of the inherent drawbacks of a dual socket system you mentioned, there's the loss of half your RAM slots and some of your PCIe when not populating a 2nd CPU, and the extra power consumption when you do populate it. Just not feeling the advantage of any of that, so I think it'll be the X10SRL for me unless someone has horror stories to share about it.

Thanks again!

 

[Magius]

Because my time, building and commissioning the server, is worth more than the money 'saved' by under speccing and building two or three times.

I couldn't agree with this more. The first server I built only lasted me ~4 years before I realized it was just not done right and not going to meet my needs anymore. I replaced it with a server that has been running strong for nearly 9 years now and still more than sufficient for my current NAS needs. The next one I build I expect to last at least as long, likely longer given how the rate of technology advancement (relative to home use) has greatly slowed.

A lot of time goes into the research, building, and eventual maintenance of these servers. I'm more than happy to spend a little bit more upfront to not have to do that all again after a few years.

 

[Stux]

I couldn't agree with this more. The first server I built only lasted me ~4 years before I realized it was just not done right and not going to meet my needs anymore. I replaced it with a server that has been running strong for nearly 9 years now and still more than sufficient for my current NAS needs. The next one I build I expect to last at least as long, likely longer given how the rate of technology advancement (relative to home use) has greatly slowed.

A lot of time goes into the research, building, and eventual maintenance of these servers. I'm more than happy to spend a little bit more upfront to not have to do that all again after a few years.

Exactly.

 

[Magius]

Well the QS Xeon from China fell through, but at least it was a painless process and a very honest seller. :(

The listing said that the CPUs were used, and "Some scratches on heatsink surface but tested working well.". The listing photo showed a picture of a CPU with light scratches on the heatsink. No big deal. that's what thermal paste is for! Unfortunately I bought the last one of 30+ (in fact, first I missed buying the last one, then it got relisted so I snagged it impulsively, now I've come to understand why...)

The next morning I had an eBay message from the seller with a picture of my CPU asking if I still wanted it. They affirmed it was still tested and working, but wanted to clear the damage with me before shipping since it was more than the 'scratches on heatsink surface' I might be expecting. I attached the pic they sent me...

Needless to say, I decided to pass on the chip, as it appears that at least five pads are shorted together. Without a pinout diagram to know what those pins are for I'm not willing to risk the hassle of an unknown CPU. So, back to scouring eBay for a cheap CPU. The only other decent deal I'm stumbled into was an E5-2685v3 QS with retail stepping for around $300-350... It doesn't support HT, but at 12 cores that shouldn't be a problem. I'm trying to keep to a hard limit of $300 on the CPU, but when my only other option is a quad-core 1620 I might be tempted to spend an extra few bucks. Fortunately I have a couple more months to play the field before I need to get this server put together.

 

[Stux]

That CPU needs to go in the bin

 

[Chris Moore]

Well the QS Xeon from China fell through, but at least it was a painless process and a very honest seller. :(

The listing said that the CPUs were used, and "Some scratches on heatsink surface but tested working well.". The listing photo showed a picture of a CPU with light scratches on the heatsink. No big deal. that's what thermal paste is for! Unfortunately I bought the last one of 30+ (in fact, first I missed buying the last one, then it got relisted so I snagged it impulsively, now I've come to understand why...)

The next morning I had an eBay message from the seller with a picture of my CPU asking if I still wanted it. They affirmed it was still tested and working, but wanted to clear the damage with me before shipping since it was more than the 'scratches on heatsink surface' I might be expecting. I attached the pic they sent me...

Needless to say, I decided to pass on the chip, as it appears that at least five pads are shorted together. Without a pinout diagram to know what those pins are for I'm not willing to risk the hassle of an unknown CPU. So, back to scouring eBay for a cheap CPU. The only other decent deal I'm stumbled into was an E5-2685v3 QS with retail stepping for around $300-350... It doesn't support HT, but at 12 cores that shouldn't be a problem. I'm trying to keep to a hard limit of $300 on the CPU, but when my only other option is a quad-core 1620 I might be tempted to spend an extra few bucks. Fortunately I have a couple more months to play the field before I need to get this server put together.

That is a pretty poor quality photo. I can't tell what that smudge is. It could be a glop of heat sync paste. If you think the seller is honest, take his word that the thing works and go for it. I have been a working full time in hardware support since 1996 and the number of CPUs that I have seen fail is probably less than 5. You can actually wash a CPU with soap and water (I have done it) and if you make sure to dry it properly it will still work. They are tough.

 

[Ericloewe]

That is a pretty poor quality photo. I can't tell what that smudge is. It could be a glop of heat sync paste. If you think the seller is honest, take his word that the thing works and go for it. I have been a working full time in hardware support since 1996 and the number of CPUs that I have seen fail is probably less than 5. You can actually wash a CPU with soap and water (I have done it) and if you make sure to dry it properly it will still work. They are tough.

I'd stick with alcohol.

 

[wblock]

It really depends on the price. And then an abused CPU could fail early for any of several reasons.

There's an E3-1230v2 on ebay right now with a corner broken off along with several pads. The description says "memory channel A is NOT working but still working with memory channel B." At $110, no, no way. And not for a NAS. At $20 for a testing system, maybe.

 

[Magius]

I appreciate the tips but I have to assume it's not just a smudge. At least the seller was up front about it, and they said something like "the metal skin has some scratches", so I assume those 5 pads have some kind of problem.

I do believe them that the cpu works fine, it's just not something I'm comfortable taking a chance on.

Saving money is one thing, but i don't want to set myself up for catastrophic failure down the road... I'll just keep my eyes open for another deal between now and the end of the year.

Sent from my SM-N900P using Tapatalk

 

[Magius]

I think I found another suitable processor, potentially even a better fit than the previous 2650. It's the E5-2618L V3, 8c/16t, 2.3GHz base with turbo to 3.4 GHz. So it's 2 less cores, the same base clock, 0.4 GHz higher max turbo, and 30W less power. Pretty much a sweet spot, I'm surprised I overlooked this model before.

Coincidentally, the seller of the 2650 that fell through has a few of these listed "used in good condition" for around the same price, so I may spring for one. This time I'll ask them for a photo first and make sure the pads are clean :) I also need to ask them about the socket, because both the listing and the HWiNFO screenshot show socket B3, LGA1356-3. That makes no sense, however, as there shouldn't be V3 Xeons compatible w/ that socket, it should be R3, LGA2011-3... Additionally, the HWiNFO screenshot shows it was taken in a Supermicro X10DAi motherboard, which is LGA2011-3, socket R3, as expected. So I'll have them clarify the socket confusion and show me a pic before ordering, but if all goes well I may have found a new CPU.