BUILD Advice and comment welcome on my proposed FreeNAS build

[cyberjock]

I don't agree. The RM4024 has these fans

http://www.chenghome.com.tw/exec/product.php?mod=show&cid=9&pid=CHD12012&lg=E

here some good photos

http://hardforum.com/showthread.php?t=1745790&page=2

A friend of mine has the Supermicro SC846BA-R920B: it is damn loud and my hard drives run 2° C hotter in his case than in mine.

I live in the South Mediterranean and temperatures and humidity here reach high levels. I tested the RM 424 in a small closet with 4 KW vented heating pushing hot air. Ambient temp was 38° C during a 9 days stress-test, with my hard disks (WD Red 3GB) running this test 24/7:

badblocks -svw -b 4096 -t 0xFF -t 0x00 -t 0xFF /dev/daX

HDs' temperature never reached 40° C. The daughterboard fan controller works well with Supermicro's BIOS, regulating the fan speed. They're not quiet, but not as loud as Supemicro's.

That proves nothing. The kind of pull you can get with an 80mm fan just can't be matched with a 120mm fan unless you start talking 100w/fan for cooling. This is strictly theoretical limitations with the physics with it. So your comparison has nothing to do with what I'm trying to explain. You can have "good" 120mm fans and "crappy" 80mm fans. But even the best 120mm fans can't compete against "good" 80mm fans. They can't.. period. The physics makes it impossible. That's why I feel confident in saying "There's a reason why the expensive case has 80mm fans and the cheap one has 120mm fans". They understand these physics.

Your drives might run 2C hotter because he's using fans with a different temperature curve. He may even have them set artificially lower RPM. So comparing 2 cases is apples to oranges. There's far more headroom to get more cooling from 80mm fans over 120mm fans.

See, since I've got a background in this kind of thing, it's easy to pick out who actually used good engineering skills in development of a product. So while you see 120mm and 80mm and have various opinions on them, I see their choices through an engineer's eyes. And I see "fail" written on cases that choose to throw 24 drives in a 4U case and then use 120mm fans.

Then, add the fact that mine were WD Green drives and imagine how much worse the problem would be with drives that use more power, or even 7200RPM drives. Can you say "nightmare"?

 

[panz]

I like the quietness of your chassis. Might get that to put my new SM motherboard in and migrate my drives over.

Moreover, the RM424 hard disk trays are quite well made and they have a little "spring loaded" (I don't know how to name it in English) to reduce hard disk vibrations. And you can control the fan speed via jumpers (see at 4:28)

View: http://youtu.be/TYpQ8Xq2ryg

 

[panz]

That proves nothing. The kind of pull you can get with an 80mm fan just can't be matched with a 120mm fan unless you start talking 100w/fan for cooling. This is strictly theoretical limitations with the physics with it. So your comparison has nothing to do with what I'm trying to explain. You can have "good" 120mm fans and "crappy" 80mm fans. But even the best 120mm fans can't compete against "good" 80mm fans. They can't.. period. The physics makes it impossible. That's why I feel confident in saying "There's a reason why the expensive case has 80mm fans and the cheap one has 120mm fans". They understand these physics.

Your drives might run 2C hotter because he's using fans with a different temperature curve. He may even have them set artificially lower RPM. So comparing 2 cases is apples to oranges. There's far more headroom to get more cooling from 80mm fans over 120mm fans.

See, since I've got a background in this kind of thing, it's easy to pick out who actually used good engineering skills in development of a product. So while you see 120mm and 80mm and have various opinions on them, I see their choices through an engineer's eyes. And I see "fail" written on cases that choose to throw 24 drives in a 4U case and then use 120mm fans.

Then, add the fact that mine were WD Green drives and imagine how much worse the problem would be with drives that use more power, or even 7200RPM drives. Can you say "nightmare"?

I have only 12 drives now, so my tests are inaccurate. I don't want to demonstrate that 120mm fans are better than 80mm ones for cooling a professional work loaded server with n. 24 7200RPM drives.

My tests are empirical and don't prove anything beside the fact that with a 38° C ambient temp for ~ 9 days all went well; neither smartctl nor IPMI showed dangerous temperatures; noise levels were good from my point of view. With the price difference between a RM424 and a Supermicro I can buy a very good (and virtually silent) Daikin AC split unit and use it during hot days.

 

[cyberjock]

And you know what? I can go buy a used Supermicro 24 drive server for $300 on ebay, throw out the crappy old hardware inside and reuse the chassis and *still* end up with a cheaper *and* better chassis than yours....

If we're going to argue about options, I can definitely beat your options. ;)

 

[trionic]

I can buy a Supermicro CSE-846BE16-R920B with single 920W 80 PLUS Platinum PSU for $1209.99 plus $63.46 international shipping:
http://www.ebay.com/itm/New-Supermi...20W-4U-Rackmount-Server-Chassis-/171136771460

That's £754.90 plus £150.98 import duty (ouch) giving a total of £905.88, just £20.28 more than the XCase RM424 Pro.

However, I just couldn't live with the noise of the Supermicro chassis. This thing has to live in an office, right next to a bedroom and will be operating 24/7.

Find me 80mm fans which are as effective as the SM fans but as quiet as the RM424 fans and I'll go with the SM case. Otherwise it has to be the RM424.

I will say though that I have had my fair share of hard disk failures over the past nine months and I am sick of it. I have two 3TB hard disks waiting for probably six weeks of dd-rescue operation. Those recent drive failures are the primary reason for building this ZFS storage chassis.

My array of Western Digital MyBooks (composed into a single FlexRAID storage pool) were great until I shifted the drives into a tower enclosure. Then they start failing even though temperatures never reach more than 30 deg C. Don't know why, other than perhaps excessive chassis vibration.

I can go buy a used Supermicro 24 drive server for $300 on ebay, throw out the crappy old hardware inside and reuse the chassis and *still* end up with a cheaper *and* better chassis than yours....

One of those (which I couldn't seem to find on EBay but I'll keep looking) plus some quiet and effedtive 80mm fans would definitely be something I'd consider.

 

[panz]

And you know what? I can go buy a used Supermicro 24 drive server for $300 on ebay, throw out the crappy old hardware inside and reuse the chassis and *still* end up with a cheaper *and* better chassis than yours....

If we're going to argue about options, I can definitely beat your options. ;)

I had both chassis in my "lab" for a few days. RM424 Pro is well built (I couldn't believe it until I disassembled both) and is clearly made with Supermicro motherboards in mind. I wasn't talking about used hardware; I never purchased it and I will never do.

One thing that make me furious about Supermicro build quality was that one of the hd trays made a lot of vibrations; this problem was due to poor machinery of the slide. Not what I expected from a $ 1600 chassis.

 

[cyberjock]

@ trionic: Those fans don't exist. You can't match the SM fans in terms of effectiveness. That's because 120mm can't compete with 80mm because of the physics. When I went fan shopping for my server I spent almost 20 hours reading about dozens of fans and looking at their fan curves to make my decision. I was trying to find a 120mm fan that "could work well enough" for my situation, but ultimately decided it was safer to just go to 80mm and deal.

As long as your fans can keep up with your heat load, everything is fine. But, as soon as you go past that breaking point, drive temps will only go up and up and up. So giving yourself as much headroom as possible to ensure you *can* keep up with your heat is the best option.

@ panz: No clue what you are talkinga bout with a tray that vibrates. Being that you are the only one I've seen to complain about tray vibration I'd have to wonder if it was a manufacturing or engineering mistake. I tend to think manufacturing since Supermicro is known for quality when it comes to building white box servers.

Sorry, but your definition of "well built" and mine aren't in agreement. Nothing personal, but when you've been an engineer you'll never look at things the same way again. You'll start questioning their choices and trying to ascertain what is or isn't a good idea regularly. There's *always* a tradeoff with every choice. As long as those tradeoffs don't significantly limit your intended use then everything is fine. But as soon as your tradeoffs become limiting(as is the case with my 120mm experience) then *someone* has made a serious mistake. In my case, just a few tests had to be done to validate that 120mm fans won't really suffice for cooling in my case without going to fans that have an extremely high RPM and draw more power than my hard drives. Sorry, but when your fans are your primary load in a server you have failed to do your job at designing a good server. ;)

 

[trionic]

@ trionic: Those fans don't exist. You can't match the SM fans in terms of effectiveness. That's because 120mm can't compete with 80mm because of the physics. When I went fan shopping for my server I spent almost 20 hours reading about dozens of fans and looking at their fan curves to make my decision. I was trying to find a 120mm fan that "could work well enough" for my situation, but ultimately decided it was safer to just go to 80mm and deal.

I get the physics around fan depth and pressure drop. I was asking about 80mm fans which retain the SM's 80mm fan effectiveness yet are quiet enough for an office environment.

 

[cyberjock]

I get the physics around fan depth and pressure drop. I was asking about 80mm fans which retain the SM's 80mm fan effectiveness yet are quiet enough for an office environment.

Right.. and I'm saying that's not going to be particularly possible... there's tradeoffs and if there were quiet fans that provided the same airflow Supermicro would probably be using them. One of their chief complaints for people that want a Supermicro case in an office environment is the fan noise.

 

[panz]

Actual power draw: 65 Watts
Ambient temp: 30° C

NAS# for i in 0 1 2 3 4 5; do
> smartctl --xall /dev/da${i} | grep -i "Current Temperature" &
> done
[1] 42005
[2] 42007
[3] 42009
[4] 42011
[5] 42013
[6] 42015
NAS# Current Temperature:               32 Celsius
Current Temperature:                    31 Celsius
Current Temperature:                    31 Celsius
Current Temperature:                    31 Celsius
Current Temperature:                    31 Celsius
Current Temperature:                    31 Celsius

NAS# sysctl -a |egrep -i -E "cpu\.[0-9]+\.temp"
dev.cpu.0.temperature: 41.0C
dev.cpu.1.temperature: 41.0C
dev.cpu.2.temperature: 42.0C
dev.cpu.3.temperature: 42.0C
dev.cpu.4.temperature: 40.0C
dev.cpu.5.temperature: 40.0C
dev.cpu.6.temperature: 36.0C
dev.cpu.7.temperature: 36.0C

IPMI screenshot

 

[ser_rhaegar]

I can go buy a used Supermicro 24 drive server for $300 on ebay, throw out the crappy old hardware inside and reuse the chassis and *still* end up with a cheaper *and* better chassis than yours....

I like the X-Case model more than the SM model based on the videos, but finding one in the US for a decent price was a no go. So SM it is. Couldn't find a SM 24 bay all week for under $600. But I did find a 16 bay which I made an offer on today for $300 out the door, rails and all. Should go well with the refurb X9SRL-F I picked up at Microcenter for $130.

 

[Lordadmiral Drake]

Cyberjock, so what would you recommend some who doesn't have a place to put a noisy server in to do?

These server fans are all damn loud. I'm glad I can put my supermicro machine in the basement

 

[panz]

Cyberjock, so what would you recommend some who doesn't have a place to put a noisy server in to do?

My intention wasn't to open a flame war about this topic, nor arguing with my friend cyberjock about what's better from an engineer perspective.

My goal was: get the best machine to do my tasks and put it in a small house with no basement and no noise insulation.

I totally agree with cyberjock that 80mm fans are the best choice if you're building a professional server for a company or the NASA or something like Amazon S3 storage purposes.

I made my choice based on these facts:

1) I couldn't find a 4U form factor case to hold more than 8 hard disks without boiling them or, as alternative, have something like an outboard marine engine running near my bedroom. So I opted for a 24 drives 4U unit with 120mm fans. They're not silent, but I can sleep while they're running at half speed in the room next to my bedroom and, yes, I'm that person who has fine ears.

2) I'm running 12 3TB WD Reds now and they're quiet, cool and well ventilated. I'm not going to put 24 7200RPM drives into my case: I purchased it to have plenty of space for putting in another 12 drives pool, do a monthly backup and pull the disks when I'm done.

This was my goal, I achieved it and I couldn't be more happy :)

 

[trionic]

What are the opinions on using the E5-1620V2 instead of the E5-2609v2?

In the UK I can get only the OEM/tray version though and so I'll have to fork out the Supermicro SNK-P0050AP4 heatsink/fan.

 

[Ericloewe]

The E5-1620 is a mighty fine processor for a very similar price. However, according to Intel, it's only available in a tray package, period, so I doubt you'll find it in retail packaging. If the price difference is as minimal as Intel suggests, go with it anyway, you get much better performance (higher clocks and more threads) when needed and use the same amount of power when you don't. The only disadvantage is that it cannot run in multiprocessor systems, but that won't be a problem here.

As for cooler suggestions, Noctua's products are very well regarded, but they're pricey (my recommendation, the NH-U9DX i4, costs some 60£). The Cooler Master Hyper 212 EVO (less than 30£) is a very popular cooler at a nice, low price point.

 

[trionic]

Thanks for the advice. I'll change the spec to include the E5-1620.

Alas the Cooler Master Hyper 212 EVO does not fit a socket 2011 motherboard. Scan have the Supermicro SNK-P0050AP4 for £33 delivered.
http://www.scan.co.uk/products/supe...0-e5-2600-series-for-x9-4u-up-dp-workstations

 

[Ericloewe]

Thanks for the advice. I'll change the spec to include the

Alas the Cooler Master Hyper 212 EVO does not fit a socket 2011 motherboard. Scan have the Supermicro SNK-P0050AP4 for £33 delivered.
http://www.scan.co.uk/products/supe...0-e5-2600-series-for-x9-4u-up-dp-workstations

It doesn't? Cooler Master's website says it does.

 

[trionic]

You're right. Apologies.

http://www.coolermaster.com/cooling/cpu-air-cooler/hyper-212-evo/

On the "Overview" tab the website says this:
"Versatile all-in-one mounting solution supporting the latest Intel® LGA 1366 / 1155 / 1150 and AMD FM2 / FM1 / AM3+."

But on the "Specs" tab it says this:
"Intel Socket: LGA 2011 / 1366 / 1156 / 1155 / 1150 / 775 * "

The Cooler Master fan looks like a great product. Good value and highly performant. Cooler Master it is! :)

 

[cyberjock]

It doesn't? Cooler Master's website says it does.

It definitely fits. I have one installed on the machine I'm using right now... ;)

 

[trionic]

It definitely fits. I have one installed on the machine I'm using right now... ;)

Would that be in a 4U case by any chance? :)