So I'm doing this a little backwards...

[FloridaDan]

In that this is my third post and not my first, but well, the world is an imperfect place.

Just wanted to stop in and say "Hey!"

My project (see signature) is well underway. Read built, running, partly configure, and being put through the ringer before I actually commit any real data to it. If I've done my math right, badblocks will finish later tonight or early tomorrow morning, and I can move on to running the extended SMART tests tomorrow night.

I'm glad I was home this weekend running badblocks, as with constant activity I was bumping up against the 40 degree mark in the late afternoons. It was both gratifying and horrifying at the same time to get the e-mail telling me that my drives where maxing out the temperature range. That prompted the current ghetto cooling scheme (cover off the case and every extra fan I have laying around plugged in and blowing into the open case.) I think when badblocks is over, I'm going to move the server to a part of the house that gets less sun in the afternoon.

Thanks to all those who have taken the time to share what they know!

 

[D4nthr4x]

Are you running the fans in the case off of the fan hub in the back? If you are you should have it turned up to max if it isn't already. I would recommend replacing the stock fans with better fans because the ones that came with the case aren't that great. I also run a fan on the passive heatsink of my atom cpu (it's just sitting on top of it). Also what are your ambient temperatures? And can you run:

Code:

smartctl -x /dev/ada0 | grep Temp

on each drive where the "0" represents the drive number.

 

[FloridaDan]

Hi there. In a word "no" about the fan "controller". Step one in my build was removing all the "unnecessary" things from the case, among those was taking the pigtail off the small PCB that the fan switch is on. I did put the switch back, as I didn't fancy a hole in the back of the case.

I have to agree about the stock Fractal fans. Strangely, two of the three fans that are part of the ghetto rig are actually stock fans that have come out of other Fractal cases. Replacing the fans in this case obviously just got bumped up in priority.

The fan that I did buy as part of the build is a 40 mm Noctua to sit on top of the passive heat sink. I just didn't like the idea of that passive heat sink in a case, that when full of drives, has very restricted airflow. I've been pleased with the temps I'm seeing out of the board with it on (consistently in the mid to upper 40's.)

Ambient temp plus the heavy load of running badblocks is where I think I got in trouble in the afternoons. The thermostat had decided that no one was home, cut the AC off, and the temperature spiked. My guess is that I made it into the high 70's in the room where the machine is. I was sitting there working, started to get uncomfortable, went downstairs to tell the thermostat I was home, came back upstairs after the AC kicked back on, and that's when I got the e-mail. I might have been OK if I wasn't running badblocks, but I'd rather not cook anything before I even get data on the system.

Thanks for the smartctl command, I've actually got a script built around something like that. I was avoiding it while badblocks is running as that operation takes so long to complete. (Running two sets of disk command from different CLIs didn't seem like a good idea.) Fortunately the system (which is admittedly probably doing something very similar in the background) worked like a charm and let me know about the problem.

 

[D4nthr4x]

So if you have the fans plugged into the mobo it's likely that they aren't spinning at full speed and are instead triggered by the temp of the CPU which does you no good when you want to cool the hard drives. There is no way that you should be getting temperature warnings with an ambient temp in the 70s I have had an ambient temp of like 86F and the hottest my drives have hit has been ~35C. So I would definitely pursue whether or not the fan headers you are using is adjusting fan speed automatically based on cpu temp and disable that feature.

 

[FloridaDan]

Fair enough, but remember that stock Fractal fans are not PWM fans, and that a 3-pin fan plugged into a 4-pin header spins at full speed regardless of what the mobo thinks it's doing with the fourth pin...

That said, and adding that I'm just an overly cautious person, I'd already turned off mobo control of the fans in bios. It's just something I do when I'm building an "always on" box, even if I know I'm plugging in 3-pin fans; the reason is that I know that something else I do is "forget" and come along later and change out a 3-pin fan for a 4-pin model and wind up in the boat you're describing.

I'll be honest, I didn't find hitting 40 C (105 F) to be that unusual given the circumstances: ambient in the high 70's, six drives, all under continual, and prolonged, stress (something like 25 straight hours of continual writes and reads), a small case, and a heck, I was even feeling a little uncomfortable (which to me is a good indication that my hardware might be feeling the same way.) I've already ordered Noctuas (yeah, I know, they're ugly, but I don't buy them for their looks) to replace the stock fans, and I'm contemplating places the server can live downstairs where the ambient temperatures stay more in the range of what's normal in a server room. (This is also an excuse to drop wired LAN to the first floor. Just sayin'. ;) ) And in the meantime watching everything very carefully; I only went the ghetto route as I didn't want to lose the test.

I am curious to hear about your fan arrangement, keeping your drives under 35 C (95 F) with an ambient of 30 C (86 F) seems impressive, but perhaps I'm thinking about this wrong.

 

[D4nthr4x]

That isn't true, motherboards will adjust voltage if PWM isn't available. Motherboards could control fan speed before PWM was a popular fan feature. I have these fans in the front: http://www.newegg.com/Product/Product.aspx?gclid=CjkKEQjw75CcBRCz2LiEs5OPsZoBEiQADgUma3k3QmSlNK04w4MhVAGv1R0b_yv8Ri_3L5oDK7T0NIPw_wcB&Item=N82E16835226039&nm_mc=KNC-GoogleAdwords&cm_mmc=KNC-GoogleAdwords-_-pla-_-Case Fans-_-N82E16835226039&ef_id=Up@G-AAABCGp9XcK:20140527183824:s but I haven't replaced my rear fan yet. I used the stock fan from the front of the case on top of my passive atom heatsink. But really I want to see if it's every drive that over-heated or if one drive got really hot for some reason which could mean that it's defective and thus I would send it back. I also likely haven't put the drives under the same stress that you have for that long of a period but I do have around 1k torrents running constantly and plex. If this was an issue for me I would setup push/pull on the hard drives and put two of them behind the hard drives.

 

[FloridaDan]

True, as I think about it there are those CHA fan headers, haven't thought about those in forever. Though regardless, I still shut the "auto-fan" or whatever they called it off in the bios.

I did sort of contemplate push-pull with the stock fans as I put the new ones in cart. It would certainly be possible to zip-tie them together and suspend the pair behind the drives. I'm going to see how the replacements do before I start seriously contemplating ghetto case mods. Even with good cable management, that extra pair of fans would make any work on the box more of a chore.

The goal of the kind of stress testing that I'm doing is to detect any week components before I actually trust them with my data, which while perhaps not earth-shatteringly important, is something that I don't want to even contemplate losing. As a result I'm going to test the system, if not exactly to failure, in a way that stresses it more than the uses to which I intend to put it. The whole burn-in is going to go something like this:

• Several passes with Memtest86 and Memtest86+ (Done. It completed four on each, no errors.)
• Long SMART test (Done. This was silly, but after I got the drives in, I just kicked this off, if I had this to do again, I'd probably do this in a different order.)
• Short SMART (Done. This was easy and painless.)
• Conveyance SMART (Done. Mostly painless.)
• Badblocks, four patterns (0xFF, 0x01, 0x10, 0xAA) Should be done now, as of the last I looked, it was about 75% of the way through the last pattern. No errors had been reported, but I'll believe that when I get home and see it with my own eyes.
• Long SMART test. Still remains to be done. I know I've already done one, but I want to make sure I didn't "wake anything up" with all the ruckuss of 40+ hours of running bablocks.

That sort of covers all the major components of the system. I'm not expecting anything to crop up at this point, but well, the best outcome of this kind of testing is to find nothing.

I'll probably add on a few simulated power outages and things like that to make sure the other safeguards are working correctly

Now I just have to figure out the POST message that I'm getting from the mobo. I'm beginning to think that it's nothing but a spurious message, but I haven't heard from anyone else who has one of these boards.

 

[D4nthr4x]

So yeah if you aren't planning on stress testing all the time you probably won't go near 40c But going overkill on cooling is never a bad idea.

 

[cyberjock]

The spec for most ATX situations claim that inlet temperature should be <80F. So being in the 70s shouldn't be "that bad". Right now my living room is 77F and my hard drives aren't screaming at me about temps(Mine are all 31-33C in my FreeNAS Mini). Granted mine are idle while yours are busy, but even for a scrub I only see a 4-5C increase at the most.

Also, *some* motherboards will adjust voltage if PWM isn't available. Also, *some* motherboards will not adjust voltage if you have it set to PWM in the BIOS. Not to razz the OP, but fan behavior is something he'd have to check the BIOS and validate for himself. I get the impression the OP has set the settings properly, but a double check wouldn't be a bad idea.

 

[FloridaDan]

No razz taken. Double-checking reality is always better than wandering off into assumption land. So as soon as I can, I will. To underscore that point, I really should have looked closer this morning, it was in the middle of the third pattern, not the fourth....the fourth is running now, so when it's done, I'll restart and go check BIOS, but until then, it's all about writin' readin', and comparin'.

And I'm off to read about POST hangs and this board.

 

[D4nthr4x]

Oh right it's an asrock, maybe your hard drives are setting themselves on fire to protest.

 

[FloridaDan]

ROTFLMAO!

While and intriguing suggestion, I'm afraid I must ultimately argue against rebellious silicone. If my drives actually are sentient and know what sort of board they're plugged into, we all have a much larger problem than my having once hit a thermal limit...

Seriously, I really did struggle with the whole idea of a full Xeon vs the Avoton (and within the Avoton class between the Supermicro and the ASRock offerings.)

The ultimate choices came down like this:

• However you slice it, a four core Xeon, Haswell or no, is going to behave like a four core chip. I have a couple, which if left on 24/7 function admirably as space heaters, even at idle. This is a bad thing in my small home office space, in Florida, in the summer. Seeing alot of positive results from the Avotons, and really wanting to ditch the space heater thing, I decided to give the Avoton a go; figuring that I could find a use for it if it turned out to be a complete bust and everything I read turned out to be just so much foo. I was betting against the "foo thing", but I needed that "other use" to get the pragmatist in me over the hump of what was basically a $400 experiment based on a few reviews and forum posts.
• The Supermicro vs ASRock thing came down to two things: UDIMM vs SODIMM and all those extra NICs:
  • If the "foo thing" came to pass and I chose ASRock I had memory I could re-purpose onto a Supermicro board for a Xeon. Total cost of "other uses" would be the cost of the ASRock board (I have some non-ECC memory I could stick in it and run it with.)
  • If I chose the Supermicro and the "foo thing" happened, well, those DIMMs were pretty much married to that board as I have no other use for 2 X 8 GB unbuffered ECC SODIMMs. Total cost of "other uses" here would be the cost of the Supermicro board plus the SODIMMs.
  • The NICs were the same way. I'm not going to team them in my environment, so having four of them didn't really matter to me. I'm just not the target market for the Supermicro board.

In the end I chose the option that for me had less of what I didn't need. Yes, that meant I had to trust a company that heretofore has really only played in the low-end enthusiast market to get a server board "right". However, all that said, folks who know this whole game way better than I currently do had already thrown their lot in with the ASRock board. So I did something that's actually pretty hard for me, and I trusted in something I couldn't see with my own eyes.

All-in-all, I'm not displeased with the choice. That's not to say that my choice is better or worse than yours, they're just different.

I am slightly miffed that I had to RMA the first board I got (that's literally the first time in like 20+ years that I've had to RMA something that had no obvious shipping damage) but hey, all things eventually come to an end.

The POST messages I've been seeing strike me as potentially sloppy coding in the BIOS (which means it could be American Megatrend's issue as much as ASRock's.) They all seem to deal with the CPU coming out of a lower power state to actually POST, or in one case cycling through the lower states and not coming out right. Neither appear to impact stability, I just prefer my world to be more neat and orderly and I do remain curious about what others see in the Event Log (in the IMPI interface, Server Health > Event Log.)


So that's way more an answer than you likely expected, but somehow it still seems "right", sorry if it's overly serious. ;)

 

[D4nthr4x]

This is why I bought the supermicro avaton (the c2758 was also cheaper than the 2750 asrock board), also SODIMMS were (and still are) cheaper on newegg for the same amount of ram.

 

[FloridaDan]

So for whatever it's worth, all the fans are set to "Full On" in BIOS.

At least babblocks is over and I can carry on with the rest of my burn-in.

 

[FloridaDan]

So for posterity, my new (Noctua) fans got here today. The difference is night and day on the HDD temps. All the drives are sitting at 31 C (that's 2 - 3 degrees lower than I was seeing with the stock fans at similar ambient temps, around 24 C), and I have much better air flow through the case. In hindsight, I probably didn't have to replace the rear fan, just the two 92 mm fans at the front. Just playing with them while I had the case open, the stock fans only moved enough air to provide a gentle breeze on my hand, even when I held my hand right behind them. The fans really move air. Right behind them, it's like a gale, with everything thing back together I can put my hand behind the HDDs and still feel the air move.

 

[chackoc]

Do you know how much of an impact the Noctua on the CPU heat sink made? Also, did you replace all of the Node's case fans with Noctuas or just a subset?

I'm trying to research the build for a quiet 6 drive system and your build seems to showcase many of the components I've been coming across. I'm curious how satisfied you are with the build overall. Is it relatively quiet in it's current incarnation?

 

[D4nthr4x]

From my experience you can put any fan on the passive heatsink to cool it. I just put the stock fans on mine and it dropped like 20c LOL.

 

[FloridaDan]

The tiny Noctua was the first fan I bought, actually with all the components. I bought it largely because of the size (and I know Noctuas to be quiet). I did not have any small fans laying around and anything over about 50 mm is going to have to rest on top of the DIMMs. I didn't really care for that idea, so I forked out $15 for the tiny Noctua. That said what D4nthr4x said. All you gotta do is get some air flow over it and you'll see your temps go down. Without the fan I was running in about the mid 50s C, with it, I'm in the low 40s C.

Regarding the other question, I replaced all three stock fans, however in hindsight, I probably only had to replace the front fans. As in all thing YMMV.

Having played with the system for a little now, I find that the front fans are where all the noise really comes from. It's a small space, and you're trying to move air through it. At the moment, I'm running my front fans at one "notch" below "full on" and it's making about as much noise as the stock fans, but the drive temps are lower. I also turned the rear fan down to about 60% I think and saw nothing "bad" happen in terms of drive temps. The CPU was unaffected.

I'm sure it would be quieter if I set the fans to PWM, but I'm not yet that brave.

As for satisfaction, over-all yes:

• The Avoton is a very capable little device (and it is shockingly small):
  • I really tried to find something wrong with it, and really can't.
  • There are some nits:
    • It doesn't have the "polish" of some Supermicro stuff I've used.
    • It whines if you don't turn off C states.
    • Oracle has really "fixed" Java. Getting into IMPI is a repetitious series of annoying clicks (but that has nothing to do with the board itself.)
  • It is not a Xeon, don't expect it to be. It is capable, and I haven't yet been able to get it to "stall".
• The memory is well, just memory. Getting memory off the ASRock QVL list is next to impossible, so I went with a recognized brand, staying away from problems (justified or no) that I'd read about.
• I like the PSU, Seasonic has always been a good brand. They put some nice touches on their products that perhaps do not make a difference in the end, but make them a pleasure to work with. That said, please note that they make a partially and fully modular models. They're hard to tell apart; you have no need for an eight-pin connector on the ASRock Avoton board. Just sayin'. I do feel that it is one of the quieter PSUs I've worked with.
• The drives. Well, they're drives. I elected to go with the the reds instead of the greens. Yeah, that cost me about $120 more in the end, but I didn't have to "do" anything to them to make them acceptable for NAS use. I am watching them as described here, for signs that the heads are "parking" too much, but that's not a big deal. Yes, I could have just run WDIDLE3 on them first, but it was honestly going to be a thorough pain to do so.
• The case. Ah, the case. OK, so let me start this by saying that I'm a bit of a Fractal Design "fan" (no pun re: the other things in this thread.) All my existing personal machines are in Fractal cases (two XLs and an R4), and when I spec "personal use builds" for those who seek my advice I invariably list a Fractal Case (unless the recipient is an incorrigible gamer or OCer who wants to see the inside of their rig.) That said, I find myself being surprisingly indifferent about the case. In the end it's fine, but I wonder if I wouldn't have been better served by the Bitfenix or the Silverstone offerings. I don't regret the choice, but it's the one thing I sort of question.

So that said, I'm sitting here probably 1.5 feet from both my FreeNAS box (it's living on the desk until I think it "settled" enough to go live in the intended space for it across the room) and a similar distance from one of my XLs that houses my office desktop. I hear the XL way more than I hear the FreeNAS box. With music playing, I can't hear the FreeNAS box unless I put my ear right up to it.

My struggles right now are with getting sharing working. ;) There's always something. I live in a "mixed" household, with both Winders and *NIX, more or less, peacefully cohabiting. Unfortunately, that "cats and dogs" thing is about to rear its head, again.

Whatever you choose, if you can afford to do so, build according to the recommended hardware guidelines. And once you've build take the time to actually test and burn-in your system. Trust me, I've done my share of "budget builds" and rushed through things because I was anxious to get a working system. Let's just say that I've learned, and that the comfort of not having to question any decisions is really worth the small premium I paid.

Also, I don't think that it gets talked about enough, but get a UPS that's just for your server. You'll be thankful sooner or later.

If I were doing this over, I would consider a DOM for a boot device.

 

[chackoc]

Sounds like it's turned out to be an excellent setup. Do you think the CPU cooler is necessary? I've usually operated under the assumption that if it's not specced with a fan then I don't need to use one.

My tentative setup was similar to yours but with some brand changes. I was looking at the fanless Seasonic PSU. I've used the fanless Seasonic in a past silent desktop build and it worked really well.

For the MB I'm still trying to research why everyone prefers the ASRock. I was going to go with the SuperMicro version since it's cheaper and has the 6 SATA ports I need, but it seems like everyone who's gone Avoton has gone ASRock.

Similar sort of issue with drives. I've always been a Seagate guy and they have the 4TB VN drives, but it seems like everyone goes with WD Reds. I need to research more about that as well.

Still, it's nice to see that the general design seems to be working well for you. I'll have to look into Bitfenix and Silverstone (haven't come across them yet) but just knowing that you've found success building a small, quiet, 6 drive FreeNas box has encouraged me that it's going to be doable.

My grail would be a completely fanless build but it doesn't look like that's technologically feasible yet. Still, a guy can dream =).

And I would definitely include a UPS. I consider them a requirement for anything other than a laptop.

I am curious about the value of the DOM though. I was under the impression that the OS will load from the boot media but that it's entirely ram resident once it's up and running. Is that not the case?

 

[D4nthr4x]

I bought the SuperMicro I have no idea why people got the asrock, it costs more and IMO asrock isn't as good of a company in general, maybe if you needed more than 6 sata ports but if you are getting this case you won't be able to use more anyway. I would highly recommend putting a fan over the CPU but it's probably not needed but running cooler generally means having a longer overall life span so I did it anyway. I wouldn't get 7200rpm drives if you want the server to be quiet because the fans will have to run at higher speed. For this reason I also went with a PSU that had a fan.