BUILD Please review my Intel Skylake FreeNAS build

[snicke]

I'm a big fan of the R5 case. I went with it for my recent build and could not be happier with it. Its quiet and cool. It's in a regular office environment and I don't ever see drive temps in excess of about 30 degrees. It comes with two 140 mm fans and I bought another one to match. Two fans in the front for intake and one in the back for exhaust. I removed the front door panel from the case for improved air flow. I considered the 804, but it looks to me like changing out a failed drive in that case would be a PITA.

Yeah, I like the idea of letting two fans in the front push air between/"through" all 8 drives placed just in front of the fans. That seems nice for the drives.

Is it i big difference regarding noise if you remove the front panel or not?

 

[jde]

No, the noise is virtually imperceptible over ambient however it's configured.

 

[snicke]

If I were to say anything, it'd be along the lines of "why Skylake", because I generally prefer not to be on the bleeding edge.

Yeah, well I guess the answer is to make the server somewhat more future proof in combination with:
* A faster and more efficient CPU
* Faster and more efficient DDR4 RAM
* The above of course gives a system with less power consumption
* Possible to increase to 64GB RAM which I probably will
* All SATA3 connections if doing something with SSDs in the future
* Most important: test Skylake so that @jgreco can use it with confidence in his next build

Sent from my Galaxy S6 Edge

 

[Kevin Horton]

* Most important: test Skylake so that @jgreco can use it with confidence in his next build

He'll want 1000 people to test Skylake before he can use it with confidence :)

 

[jgreco]

He'll want 1000 people to test Skylake before he can use it with confidence :)

Yes, it's always nicer to have someone else be the guinea pig.

Bent pins, design errors, etc.

Honestly, the generation-to-generation improvement of CPU's since Sandy Bridge are quite modest. Let's review the last 5 years.

Sandy Bridge E3-1230 v1 - 3.2GHZ, 8MB - Geekbench ~11000

Ivy Bridge E3-1230 v2 - 3.3 GHz, 8MB - Geekbench ~12000 - note MHz increase

Haswell E3-1230 v3 - 3.3 GHz, 8MB - Geekbench ~13000

Haswell Refresh E3-1231 v3 - 3.4 GHz, 8MB - Geekbench ~13500 - note MHz increase

Broadwell v4 - effectively skipped(!)

Skylake E3-1230 v5 ~ 3.4 GHz, 8MB - Geekbench ~13000

All of these CPU's are effectively in the "entry level" ~$220-240 price range for the Xeon, so I deem them approximately equivalent.

My perspective is that there haven't been any significant changes since Sandy, except for some gradual evolutionary improvements, and finally with Skylake, the big RAM jump. These new parts do not consume "significantly" less power for meaningful values of the word; I benched a complete small Sandy system idling at 44 watts in 2011.

When you start to consider that, then all of a sudden it does not seem so urgent to jump on the "latest and greatest" bandwagon.

 

[Dice]

When you start to consider that, then all of a sudden it does not seem so urgent to jump on the "latest and greatest" bandwagon.

Kind of doubting my choices already (also ordered a skylake setup a week ago, still in the mail).
As from a windows home user perspective, it just does not make any sense what so ever to go for other than the latest technology.
The count of times I've slapped myself around /with a large trout, due to applying <windows user logics> to assumed functionalities of FreeNAS is.. well... ..at least on decline :P

 

[jgreco]

The beautiful thing about having a place to talk about it all is that you can gain other perspectives and other opinions. And around here, they're usually pretty competent.

 

[schoffman]

The reason I am almost ready to pull the trigger on skylake instead of haswell is because DDR4 is runs at a lower voltage then DDR3 so therefore it is probably more efficient, you get double the memory capacity with skylake, 8 sata ports vs 6 ports without HBA.

To the OP I don't understand why you want the HBA right out of the gate, for me with a very similar use scenario I think I will do 7x4TB in a Z2 with the 1 extra port for boot ssd. Your power supply only has 8 sata connectors so I don't know how you will get more. Your case can only holds 8x3.5 bays and I'm unsure if 3.5 disks in the 5.25" bays will get enough cooling. It seems like the HBA only buys you one extra 3.5 disk and the cost benefit ratio seems steep.

 

[jgreco]

"probably more efficient" in the same extremely marginal way that lots of things are slightly more efficient, nothing that'll change your life though.

I really can't tell the difference when a VM is running on one of our latest generation DDR4-2133 based boxes or an old Sandy DDR3-1333. By far the biggest accelerant is the use of a high quality disk subsystem.

 

[schoffman]

It is my understanding that DDR4 was designed from the ground up to be more efficient then DDR3, 1.2 volts vs 1.5 volts.
Comparing a Haswell vs Skylake Build I get the following:
X10SLH-F+e3-1231v3+32 gigs crucial ecc memory: $690 (newegg and crucial vendor)
X11SSM-F+e3-1230v5+32 gig (2*16) of rec Samsung memory:$687 (newegg and super micro estore)

So same price for Skylake with 2 extra sata ports and ability to upgrade to 64 gb in the future. You can see why someone would have a hard time picking Haswell over Skylake.

 

[snicke]

To the OP I don't understand why you want the HBA right out of the gate, for me with a very similar use scenario I think I will do 7x4TB in a Z2 with the 1 extra port for boot ssd. Your power supply only has 8 sata connectors so I don't know how you will get more. Your case can only holds 8x3.5 bays and I'm unsure if 3.5 disks in the 5.25" bays will get enough cooling. It seems like the HBA only buys you one extra 3.5 disk and the cost benefit ratio seems steep.

* I want RAIDZ3 but I need the storage an don't want larger, more expensive and less reliable drives (6TB). Hence 8 3.5" drives connected to the motherboard.
* 8 Sata default from the PSU is plenty. When you need more you solve it easily with one or several of these: http://m.newegg.com/Product/index?itemnumber=12-200-468 or a molex-to-sata cable directly from the PSU or both. 8 is not a limit.
* The case can hold 8 3,5" + 2 SSDs, and I will use 1 SSD for boot and 1 (or 2 in mirror) for jails for performance. 1 extra SSD kan always be mounted. The 5.25 case can be removed and you can mount an SSD there with adapter (and maybe some creativity if needed). Hence I will have up to 3 2.5 SSDs connected to the HBA.
* I've got the HBA for a nice price. Around $120 shipping included wirh rapid delivery within Europe.


Sent from my Galaxy S6 Edge

 

[Bidule0hm]

you solve it easily with one or several of these: http://m.newegg.com/Product/index?itemnumber=12-200-468

No! never use one of those SATA power splitters, they are fire waiting to happen in your server... Molex to SATA splitters are fine however (molex connectors can handle a lot more current than SATA power connectors, that's why) ;)

 

[Ericloewe]

No! never use one of those SATA power splitters, they are fire waiting to happen in your server... Molex to SATA splitters are fine however (molex connectors can handle a lot more current than SATA power connectors, that's why) ;)

To be fair, that's only a problem with El Cheapos. That said, it's harder to get the molexes wrong, since the terminals are crimped pins.

 

[TheKiwi]

I can't trust SATA power splitters/adapters at all. There's so many cases of them horribly failing and catching fire. I suspect there's at least one major OEM out there who has been releasing completely flawed adapters and has basically "poisoned the pool", so to speak.

 

[jgreco]

To be fair, that's only a problem with El Cheapos. That said, it's harder to get the molexes wrong, since the terminals are crimped pins.

Yes, and telling the difference between El Cheapos and El Cheapos that have been marked up by opportunistic price gougers to be El Expensive-o-el-cheapos is very hard. Suggesting that this is at all an option is a bad idea.

Refer to https://forums.freenas.org/index.php?threads/first-freenas-build-verification.38807/#post-237362

 

[snicke]

No! never use one of those SATA power splitters, they are fire waiting to happen in your server... Molex to SATA splitters are fine however

Well, as always it is really important that your cables (and connectors!) can handle the current passing through them. Therefore it's actually important (but not easy to judge) to take care in finding quality cables/connectors an don't cascade connect several of them. But I wouldn't be that sure that "molex to sata" is fine or even better to use than a molex to sata connector. There are a lot of stories like these on the web: http://www.stevenhale.co.uk/main/2012/05/sata-plug-melted/ Google for a lot more examples against molex-to-sata adapter. In fact the molex connector has a reputation to being unreliable since the "spring connectors" can become "lose" over time and you could get a bad physical connection that also has caused fires by a lot of current passing a very small are of the connector creating a lot of heat (Google it. But then again, on the web everything has caught fire somewhere. I bet even a glass of water at some corner of the planet).

But to broader the discussion. Are there any other alternatives besides Sata splitters and molex-to-sata adapters? I haven't found any PSU around 600 W with that many "native" sata power connectors. 8 seems like a lot sata power connections in that class. Or am I wrong?

So what is the best practice (concentrating on powering the drives) if you want to use a 850 W PSU and connect e.g. 16 sata drives? How have you solved the problem in your systems?

Sent from my Galaxy S6 Edge

 

[Bidule0hm]

The molex connector was designed to handle more current than the SATA connector so it's safer to use a molex to SATA splitter than a SATA to SATA splitter.

Are there any other alternatives besides Sata splitters and molex-to-sata adapters?

If your PSU is modular then you can order SATA cables to use in place of the molex cables ;)

 

[snicke]

The molex connector was designed to handle more current than the SATA connector so it's safer to use a molex to SATA splitter than a SATA to SATA splitter.

Yes. I'm just saying that with bad quality cables and/or bad physical connection that solution can also be dangerous. However, I would go with a molex-to-SATA anyway instead of a SATA splitter. When answering @schoffman above I mainly wanted to say that the number of included SATA connectors to a PSU is not a limit for the number of possible SATA connectors for that PSU and then linked to a random solution for extending the possibilities.

If your PSU is modular then you can order SATA cables to use in place of the molex cables ;)

That sounds like the best solution. Then they at least are of the same quality as the ones included. I will check that with Corsair for the RM650i. Thanks.

 

[jgreco]

I bet even a glass of water at some corner of the planet

West Virginia.

But to broader the discussion. Are there any other alternatives besides Sata splitters and molex-to-sata adapters? I haven't found any PSU around 600 W with that many "native" sata power connectors. 8 seems like a lot sata power connections in that class. Or am I wrong?

Previously linked to the answer: https://forums.freenas.org/index.php?threads/first-freenas-build-verification.38807/#post-237362

"Find a friendly electronics shop, buy them a case of beer, and they'll probably be happy to make you a decent wiring harness that doesn't involve bodged together connectors."

Basically you just need someone with the tooling and consumables to make cables. They don't necessarily even need to have your specific PSU's "modular" connectors, since those can be snip-and-splice. Actually the whole thing could be snip-and-splice, since you can order nice quality molded SATA power connectors (think: Molex-to-SATA). But lots of us have the actual parts to do a full wiring harness.

 

[Bidule0hm]

Yep, personally I think I'd just use the soldering iron to solve this problem... :)