BUILD Sanity Check X10 box

[mrd]

Hi everybody,

Been lurking around the forums for the past year, biding my time (and money) for a FreeNAS build. I'm now prepared to take the dive and would be very grateful if you could take a look at my part-list. It's pretty mundane, to be sure:

PSU: Seasonic G-450
MB: SUPERMICRO X10SLH-F-O XEON3 INTEL C226 UA
CPU: Xeon e3-1231 v3
RAM: 2x Crucial CT2KIT102472BD160B 16GB (2x 8GB) Memory Kit (Total: 32GB)
HDD: 6x HGST 4TB Ultrastar 7K4000 (HUS724040ALA640)
UPS: CyberPower CP1500EPFCLCD (900W)
CASE: Fractal Design Node 804 mATX
BOOT: cheapest MLC SSDs I can find at the time of purchase

Most are bought from Amazon.de/co.uk. The UPS and HDDs I already own (got a huge deal on the HDD).

Purpose:
- general backup for family photos/videos
- all of my (legally owned) music collection, ripped to flac (pretty big)
- e-book library
- backup for a dozen or so VMs (3 snapshots/VM)
- streaming FHD movies to my parents/in-laws and my own phone when I'm away (got 300/200 Mbit Internet connection)
- ownCloud
- 5+ years life-time

Besides any glaring mistakes with the build above, I'd like to ask the following:
1. Is the PSU sufficient? According to jgreco's thread on sizing, it should be.
2. Stress tested the HDDs when I bought them according to the HDD Burn-in thread. Each passed, but I'm a bit concerned about the temps: with a one-side open Fractal R4 I got 47C max on pretty much every drive during the bad-blocks test. Should I go with this case (node 804) or is another more suitable (preferably from Fractal Design) ?
3. If I stick with the case from above, which fans should I go for? Keep stock? Go for Noctua AF-14? PWM?
4. For ZFS I'm planning on a single raidZ2 config. Optimal for 6 disks, right?

Thanks!

P.S. I was also considering an X11 build, but decided against, as it would have been ~150E more expansive than this one (and what's wrong with sticking with something tried-and-true?).

 

[Robert Trevellyan]

I'm a bit concerned about the temps

With good reason.

with a one-side open Fractal R4 I got 47C max on pretty much every drive during the bad-blocks test

What do you mean by "one-side open"?

 

[Robert Trevellyan]

I'm planning on a single raidZ2 config. Optimal for 6 disks, right?

RAIDZ2 with 6 disks is a very popular configuration, and a great compromise between reliability, capacity and cost. There is no "optimal".

 

[Sir_Slappy]

I got 47C max on pretty much every drive during the bad-blocks test. Should I go with this case (node 804) or is another more suitable (preferably from Fractal Design) ?
3. If I stick with the case from above, which fans should I go for? Keep stock? Go for Noctua AF-14? PWM?

Yeah those HDD temps are scary high. Best to try and keep them under 40C at the absolute minimum but the lower the better. The fans that come stock on the Node 804 (Silent Series R2) are nothing to write home about and if possible I would replace them with higher performing fans. Depending on your tolerance to high fan speeds and where the unit would be located, I would check out the Noctua IndustrialPPC fans myself. They offer higher RPM/higher CFM than the regular Noctua fans, have 6 year warranties, do not have the puke brown color, and also use very low amounts of wattage for the performance that they offer. Their static pressure is very high for their size so they would do a good job of forcing air through the cramped HDD cages to help keep them cool. Or you could always buy some Delta's...

 

[mrd]

With good reason.

What do you mean by "one-side open"?

As in the side panel taken off. Though I misremembered in the first post and the high temps were with the side panel on. Off and they were 39-41C.

 

[mrd]

Yeah those HDD temps are scary high. Best to try and keep them under 40C at the absolute minimum but the lower the better. The fans that come stock on the Node 804 (Silent Series R2) are nothing to write home about and if possible I would replace them with higher performing fans. Depending on your tolerance to high fan speeds and where the unit would be located, I would check out the Noctua IndustrialPPC fans myself. They offer higher RPM/higher CFM than the regular Noctua fans, have 6 year warranties, do not have the puke brown color, and also use very low amounts of wattage for the performance that they offer. Their static pressure is very high for their size so they would do a good job of forcing air through the cramped HDD cages to help keep them cool. Or you could always buy some Delta's...

Sorry, but could you please tell me what Deltas are?
Regarding noise, I plan to put the box in a closet, so I don't mind if it gets loud, just as long as I don't need to soundproof the closet door.

P.S. Is there any multi-quote option? I hate cluttering the thread with a separate post each time I want to quote someone.

 

[Robert Trevellyan]

As in the side panel taken off.

Do not run your system with the side panel off for any length of time, the airflow will be all wrong.

Is there any multi-quote option?

Select the text you want to quote and click the little Reply button that appears attached to it.

 

[Sir_Slappy]

Sorry, but could you please tell me what Deltas are?
Regarding noise, I plan to put the box in a closet, so I don't mind if it gets loud, just as long as I don't need to soundproof the closet door.

http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=100007998 50001445&IsNodeId=1 These are Delta fans. They are pure server grade, no-frills fans that at excel at one thing...moving obscene amounts of air with complete disregard to noise emitted while doing so. Seriously, they are extremely high rpm and extremely loud. They are extremely well made (just like most of their PSU's) and last a very long time. For your situation though I think they would be overkill. Due to the power requirements for their fans I would just stick with Noctua fans myself unless I absolutely needed 5,000 rpm fans for some odd reason

 

[Steve Brown]

Your configuration is good but the reason those drives are probably running so hot is that they are 7200 RPM also 3.5" HDD's. I would go to 5400 and 2.5" which would bring the temperature down. For the purposes you use it for the bottle neck will not be the HDD's it will be your LAN/WAN connection. 6 drives at 5400 RPM should pull some very good numbers. I am very close to your setup (look at my sig).

Here are my write tests... at different times of the day.

[root@freenas] ~# dd if=/dev/zero of=./test bs=4M count=1k
1024+0 records in
1024+0 records out
4294967296 bytes transferred in 0.726527 secs (5911641880 bytes/sec)

 

[Kevin Horton]

I've got a Node 804 case, currently with 5x4TB WD Red drives (5900 rpm). The drive temperatures approached 40 deg C during the multi-day long badblocks tests, but they are usually running around 37 deg C in service, with the fans at minimum speed.

I replaced two of the supplied fans with cheap 4 pin PWM (i.e. 4 pin connector) fans, so the motherboard could control fan speed. I set up a script that runs every three minutes, and adjusts the fan speed as a function of the hottest drive temperature - the fans are running at minimum speed 99% of the time, but they do speed up if the temperature starts to rise. My in-service drive temperatures have never exceeded 39 deg C.

I think you will be OK with drive temperatures with the Node 804 case, but you may need to experiment to find the best number and placement of the fans (the case can accept up to 5 fans on the drive bay - I'm only using 2, and they are at minimum speed).

 

[mrd]

Your configuration is good but the reason those drives are probably running so hot is that they are 7200 RPM also 3.5" HDD's. I would go to 5400 and 2.5" which would bring the temperature down. For the purposes you use it for the bottle neck will not be the HDD's it will be your LAN/WAN connection. 6 drives at 5400 RPM should pull some very good numbers. I am very close to your setup (look at my sig).

Here are my write tests... at different times of the day.

[root@freenas] ~# dd if=/dev/zero of=./test bs=4M count=1k
1024+0 records in
1024+0 records out
4294967296 bytes transferred in 0.726527 secs (5911641880 bytes/sec)

You're right ofc, but as I said I got them on a really good deal (half street price) from a local start-up that went belly-up, so I can't give them back. I guess I'll need to invest some more cash in better fans.
Also, regarding those tests, you might want to add the fsync flag to dd, to make sure the data is also written to disk and not cached in the kernel (not sure if required for freebsd/zfs, but that's best practice on linux/ext*). Nice rig, drooling after that NIC :).

These are Delta fans. They are pure server grade, no-frills fans that at excel at one thing...moving obscene amounts of air with complete disregard to noise emitted while doing so. Seriously, they are extremely high rpm and extremely loud. They are extremely well made (just like most of their PSU's) and last a very long time. For your situation though I think they would be overkill. Due to the power requirements for their fans I would just stick with Noctua fans myself unless I absolutely needed 5,000 rpm fans for some odd reason

I don't think those are even available in Europe (they don't show up in any of my go-to shops/amazon.de), but I agree they're probably overkill. Will look into the Noctuas, thanks for the advice.

I've got a Node 804 case, currently with 5x4TB WD Red drives (5900 rpm). The drive temperatures approached 40 deg C during the multi-day long badblocks tests, but they are usually running around 37 deg C in service, with the fans at minimum speed.

Those are pretty much the temps I'm aiming for. How's the noise?

I replaced two of the supplied fans with cheap 4 pin PWM (i.e. 4 pin connector) fans, so the motherboard could control fan speed. I set up a script that runs every three minutes, and adjusts the fan speed as a function of the hottest drive temperature - the fans are running at minimum speed 99% of the time, but they do speed up if the temperature starts to rise. My in-service drive temperatures have never exceeded 39 deg C.

Thank you for the link, I read that thread some time ago and couldn't remember its name. Which fans did you replace exactly?

Thanks again for taking the time guys, you're great!

 

[Kevin Horton]

Those are pretty much the temps I'm aiming for. How's the noise?


Thank you for the link, I read that thread some time ago and couldn't remember its name. Which fans did you replace exactly?

The noise is not an issue for me. The NAS is not completely silent, but it is quite quiet, as long as the fans are running at low speed (which is 99% of the time). I can hear it, if I actively listen for it, when the fans are on medium speed (my script sets maximum speed if the hottest hard drive is 40 deg C or warmer, medium speed if the temperature is 39 deg C, and minimum speed if the temperature is 38 deg C or cooler). I've got FreeNAS setup to email me if any drive temperature hits 41 deg C - I have only ever received an email when I temporarily set the threshold lower for testing. My hard drive control script also logs the temperatures

The Fractal Design fans that come with this case are three wire fans, that cannot be controlled by my SuperMicro X10SL7-F motherboard. The fans run at max speed, if you plug them into the mother board. The case has a fan controller to power the fans, and a switch that lets you manually select low, medium or high speed. I've got one Fractal Design fan on the motherboard side of the case, plugged into the fan controller on the case, running at low speed all the time. I've got two DeepCool UF-120 fans on the hard drive side plugged into the motherboard - one running as an inlet fan and one running as an exhaust fan. These cheap fans are quite quiet and work perfectly fine for my mission.

Here are the current temperatures and fan speeds:

Code:

CPU 0: 33 C
CPU 1: 33 C

da0 WD-WCC4E4TSK08F: 32 C
da1 WD-WCC4E4TSK1YA: 36 C
da2 WD-WCC4E2VTA4VZ: 34 C
da3 WD-WCC4E1HHLPL7: 35 C
da4 WD-WCC4E4TSK09J: 34 C

FAN1             | 1700 RPM          | ok
FAN2             | 400 RPM           | ok
FAN3             | 400 RPM           | ok

• FAN1 is the CPU cooler,
• FAN2 & 3 are the DeepCool fans,
• the Fractal Design fan is not hooked up to the motherboard, so it is not monitored.

 

[ChriZ]

@mrd , one thing I noticed in your first post is that you will be using 6 hdds and you are planning for cheap ssds for boot.
However, your motherboard has only 6 sata ports...

 

[Steve Brown]

MRD can purchase a M1015 for under 100 off ebay and be golden!

 

[Steve Brown]

you might want to add the fsync flag to dd

Yeah I tried conv=fsync and conv=fdatasync and no luck so I am guessing it was done right so maybe those are just my awesome numbers :)

 

[mrd]

@mrd , one thing I noticed in your first post is that you will be using 6 hdds and you are planning for cheap ssds for boot.
However, your motherboard has only 6 sata ports...

That's the kind of obvious mistake I was hoping someone would catch. Thanks!

Solutions to this:
1. LSI expansion card
2. different mobo
3. boot off USB sticks - though with the recent change to ZFS on boot devices doesn't sound so good
4. sata DOM - kind of expansive compared to usual SSDs

Which would be best from a price/perf perspective? Going for USB sticks is cheapest, then sata DOMs and then different mobo. Or is there another option?

Yeah I tried conv=fsync and conv=fdatasync and no luck so I am guessing it was done right so maybe those are just my awesome numbers :)

No luck, as in dd returns an error?
You've 5.5 GB/s speeds there. Could someone else confirm such write speeds?

MRD can purchase a M1015 for under 100 off ebay and be golden!

Yeah, under 100 in the US, but over here in euro land cheapest I can find is 150E (~165$).
Other options include x10sae with 2 additional ASM sata ports, which don't inspire much confidence but isn't much more expensive, while the x10sra with 10 ports sets me back another 140E plus.
Too bad I can't use X10sl7's sas ports...

 

[Linkman]

Too bad I can't use X10sl7's sas ports...

Why not? You can plug the usual SATA drives into those ports.

 

[mrd]

Why not? You can plug the usual SATA drives into those ports.

Just checked and you're right. I was under the impression that SAS ports don't work with SATA drives, but it's the other way around. Well, that solves it then. I'll go for the X10sl7.
Is there any difference where I plug my drives in? Like boot drives should preferably go into the SATA ports, or smth like that?

 

[Linkman]

I think I've read that the ports on the LSI controller might be a bit faster? So I put my main pool on those ports, and am putting a second mirrored pool and my boot SSD on the motherboard ports. Otherwise, I don't think it matters.

 

[BigDave]

Regarding noise, I plan to put the box in a closet, so I don't mind if it gets loud, just as long as I don't need to soundproof the closet door.

If you don't provide fresh cool air and hot air escape with some type of forced air or venting system, your machine
will burn up, but it will, in short order, be VERY quiet...