Building new vs. buying an old server

[Dice]

Here's an anecdote of accidentially 'introducing' my SM box to a friend:
My buddy came over the first day I had the box temporarily powered up in my living room.. Messing around with IPMI and BIOS settings.
I live on the 4th floor.
Upon entering the 'grand door' to the staircase, he immediately identified "oh - someone is vacuuming".. Once he arrived at the 3th floor, he'd said he was like "oh - this has to be the appartment that is vaccuming"... only to realize it was coming from the floor above.......
His jaw dropped quite a bit when I <opened> my door and he heard the ..NOISE LEVEL in my appartment. :P :P

This definitely is no exaggeration of the event that happened.
Needless to say, none of us had much experience with true server gear before.
He'd probably never ever consider getting a server box to put in an appartment x)

(I've not given up yet, I'm considering to make a real effort to deadening/isolating sound before deciding if that SM 36bay box has any business being remotely close to a habited area of mine)

 

[Ericloewe]

Yeah, only more recent PSUs seem to have temperature-controlled fans. Older models have their bazillion-k RPM 40mm fans at full blast 24/7.

 

[taylornate]

I'm completely comfortable with the idea of replacing fans in the chassis and power supplies.

Still thinking on modern vs. old for everything but the chassis.

 

[Mirfster]

Go with the older stuff... Its cheaper and you'll be "Hip"... ;)

/At least that is what I keep telling myself about my Servers...

 

[taylornate]

Well, there's this system, which costs only $15 more than I was planning on spending on the chassis alone.

CPU: 2x E5520 2.26ghz quad core nehalem
motherboard: X8DT6-F which has LSI SAS 2008 on-board, which looks compatible with freenas according to this thread.
memory 8gb DDR3... will need more of that, obviously

There are some variations on this with more memory and/or with the X5650 2.66 ghz 6-core westmere. The price obviously goes up but... yeah... leaning back toward old.

 

[Mirfster]

Alright, well I will toss this into the mix to further confuse you...

Here is my posting on my favorite system (let the others begin their moaning now... ;) )

Also, I have convinced a seller to create a "FreeNAS Turn-Key" setup that basically gets you there (minus the hard drives) for ~$400.00 all in. You can see that info on Post #28 in the thread. I am just awaiting delivery of the system so I can do a full review.

 

[maglin]

I can vouch that it's a pretty quite server enclosure. I want another one in the future.


Sent from my iPhone using Tapatalk

 

[Dice]

I'm completely comfortable with the idea of replacing fans in the chassis and power supplies.

Still thinking on modern vs. old for everything but the chassis.

The fans are loud for design reasons, not by poor quality.
Which is the case in most consumer grade workstations, allowing replacement of fans to higher quality models to be a successful operation.
Therefore, it is misleading to attempt fan replacement in servers.

I did some elaborations to attempt to find a more suitable noiselevel in my 2xL5630, 36bay box.
One of the more important capabilities of the included fans, distinguishing them from desktop versions, is the capacity to provide high air flow (see link in my sig) in spite of being heavily unfavorable air pressure in the box.

My chassis has additional intakes on the sides, behind the backplane. The purpose of these is probably to releif a tad of pressure from a fully populated front, to provide accurate cooling to much higher TDP CPU's than the L5630 I've in my machine. This is a possibility for responsible exploitation:

I covered these intakes on both sides, and covered the rear 12 bays by removing the trays and covering the area with tape/card board.
This did a humongous difference in airpressure in the area where it is needed (<24bays populated).

The other mod I did was to move fanheaders from the backplane to the motherboard. The backplane do not allow for easy adjusting of fan speeds through IPMI, nor the BIOS settings for different cooling profiles.
Instead I plugged the fans into the motherboard directly, being cautions of not overloading the maxrating of the motherboards capacity (these are heavy duty fans - and fully populating all fan headers can indeed lead to hardware failure).
Fortunately, I was satisfied with trying with no more than 3 fans, for my first heat and noise evaluation.

Finally - removing the front 4 out of the 7 fans in the midplane:

I found the CPU's ran a few degress hotter at max load (60c), still within acceptable values.
Slightly worse though, I did notice some increased PSU temperature assumingly leading to unneccessary noise. Even if the PSU registered about 240w pulled power and a tempearture of 24c you'd imagine they'd be rather gentle on the fan speed? NO!
This find encouraged me to reconnect the fan in front of the PSU, to the left in the picture. The situation improved.

After these mods, drive temps (and CPU) read the same level as when all fans went on full retard hurricane-mode, during regular usage and high IO mode.
As indicated by my previous post, I'm still a long way from fully adopting this monster.
(I might add I'm a bit 'allergic' to unwanted noise - my main rig runs completely passive with the exception of full load. My screen litterary makes more noise.)

Hope this was helpful :)

 

[Valdhor]

Well, there's this system, which costs only $15 more than I was planning on spending on the chassis alone.

You may want to ask the seller for pictures of the actual box. The picture in the listing is not what the listing specifies.

 

[Dice]

You may want to ask the seller for pictures of the actual box. The picture in the listing is not what the listing specifies.

I'm curious here, what was the discrepancy? Looks the same to me: https://www.supermicro.com/products/chassis/4U/846/SC846E16-R1200.cfm

 

[Valdhor]

The listing specifies 2 x Intel Xeon E5520 - The picture has no CPU's nor heatsinks. Also specifies an Adaptec card (Not that you'd want to use it) and there are no cards in the picture. Do the SAS cables shown in the picture come with it? How would you know as that is not a picture of what the listing says. There is more RAM installed than what the listing says - I see three sticks; The listing specifies two.

I like to see what I'm actually ordering if it's used. If it's something new I don't mind as much. YMMV.

 

[Dice]

@Valdhor right. I thought your concern was related to the chassis itself.
From my ebaying around, used servers from 'recycling companies' very rarely show the actual box /w intestants.

 

[Valdhor]

That's why I always ask the seller for pictures of the actual box with innards. If they won't do it I don't buy from them. There are a few that will do that for you.

 

[taylornate]

Welp... I pulled the trigger on this 36 bay system.

 

[Valdhor]

That'll work. Hope you're ready for the 747 takeoff noise ;)

 

[taylornate]

That'll work. Hope you're ready for the 747 takeoff noise ;)

If I can hear it from my basement, I'll make the modifications that Dice outlined earlier.

 

[Dice]

If I can hear it from my basement, I'll make the modifications that Dice outlined earlier.

Depends on the thickness of your floor ;)

Just be cautious of the modifications suggested. They are no recepie for success and should not be seen as a "guide to fix the noise" but rather as a data point in what results my modifications yeilded in my particular case.
I've seen others having outragous temperatures in their 847's resulting from non successful modifications or simply bad server environment.

I strongly suggest you keep a very close eye on temperatures. CPU as well as drives. CPU - particularly after a couple of hours stresstesting when the materials have equalized heat across mass. Same for HDD's obviously.
Also: My case was not fully populated, at the time for testing.
Good luck :)

 

[taylornate]

The fans are loud for design reasons, not by poor quality.
Which is the case in most consumer grade workstations, allowing replacement of fans to higher quality models to be a successful operation.
Therefore, it is misleading to attempt fan replacement in servers.

I did some elaborations to attempt to find a more suitable noiselevel in my 2xL5630, 36bay box.
One of the more important capabilities of the included fans, distinguishing them from desktop versions, is the capacity to provide high air flow (see link in my sig) in spite of being heavily unfavorable air pressure in the box.

My chassis has additional intakes on the sides, behind the backplane. The purpose of these is probably to releif a tad of pressure from a fully populated front, to provide accurate cooling to much higher TDP CPU's than the L5630 I've in my machine. This is a possibility for responsible exploitation:

I covered these intakes on both sides, and covered the rear 12 bays by removing the trays and covering the area with tape/card board.
This did a humongous difference in airpressure in the area where it is needed (<24bays populated).

The other mod I did was to move fanheaders from the backplane to the motherboard. The backplane do not allow for easy adjusting of fan speeds through IPMI, nor the BIOS settings for different cooling profiles.
Instead I plugged the fans into the motherboard directly, being cautions of not overloading the maxrating of the motherboards capacity (these are heavy duty fans - and fully populating all fan headers can indeed lead to hardware failure).
Fortunately, I was satisfied with trying with no more than 3 fans, for my first heat and noise evaluation.

Finally - removing the front 4 out of the 7 fans in the midplane:

I found the CPU's ran a few degress hotter at max load (60c), still within acceptable values.
Slightly worse though, I did notice some increased PSU temperature assumingly leading to unneccessary noise. Even if the PSU registered about 240w pulled power and a tempearture of 24c you'd imagine they'd be rather gentle on the fan speed? NO!
This find encouraged me to reconnect the fan in front of the PSU, to the left in the picture. The situation improved.

After these mods, drive temps (and CPU) read the same level as when all fans went on full retard hurricane-mode, during regular usage and high IO mode.
As indicated by my previous post, I'm still a long way from fully adopting this monster.
(I might add I'm a bit 'allergic' to unwanted noise - my main rig runs completely passive with the exception of full load. My screen litterary makes more noise.)

Hope this was helpful :)

Wow, this cut way down on the noise, especially at low load. I can't believe those 3 fans were not under any kind of control. Temps are much higher under max load, up to 81C. I'd like to know what the difference is between this and your temp of 60C. A few ideas:

1) Settings - My system thinks 79C is the cutoff for 'high', and 89C is 'critical'. I'm using the most conservative fan setting in the bios.
2) Do you have fans attached directly to the CPU heat sinks? I don't.
3) What are you calling "max load"? I'm running a stress test using mprime.

Do any of these strike you as a possible difference?

edit:
Oh, and an update...

I found a deal on used enterprise-grade HGST 2TB drives and bought 15 of them. One will be a hot spare. They will run as mirrored pairs alongside my original pair of 4TB HGST drives for a total of 18TB of usable space. I have a pair Kingston 32GB SSDs that I will use for the OS. I also picked up 2xAPC 2200 VA UPS units, and ran 2x 20 amp circuits on separate poles. I'll run the stress test overnight and then I suppose I will be ready to install freenas.

I haven't picked up a rack-mount switch yet. Any suggestions? I'd like to hold out for one with a couple 10GB SFP+ ports, but I'm not sure if it's worth it. Any thoughts on that?

 

[taylornate]

Wow, this cut way down on the noise, especially at low load. I can't believe those 3 fans were not under any kind of control. Temps are much higher under max load, up to 81C. I'd like to know what the difference is between this and your temp of 60C. A few ideas:

1) Settings - My system thinks 79C is the cutoff for 'high', and 89C is 'critical'. I'm using the most conservative fan setting in the bios.
2) Do you have fans attached directly to the CPU heat sinks? I don't.
3) What are you calling "max load"? I'm running a stress test using mprime.

Do any of these strike you as a possible difference?

edit:
Oh, and an update...

I found a deal on used enterprise-grade HGST 2TB drives and bought 15 of them. One will be a hot spare. They will run as mirrored pairs alongside my original pair of 4TB HGST drives for a total of 18TB of usable space. I have a pair Kingston 32GB SSDs that I will use for the OS. I also picked up 2xAPC 2200 VA UPS units, and ran 2x 20 amp circuits on separate poles. I'll run the stress test overnight and then I suppose I will be ready to install freenas.

I haven't picked up a rack-mount switch yet. Any suggestions? I'd like to hold out for one with a couple 10GB SFP+ ports, but I'm not sure if it's worth it. Any thoughts on that?

edit:
The Blade G8000F seems to be pretty reasonably priced on ebay. Anyone know anything about it?

 

[cryptyk]

I just bought a used wd arkeia RA6300 on ebay for $1,600. It cost about $45,000 brand new just a couple of years ago. I couldn't even buy the 48TB of drives that it came with for $1,600.
I was totally convinced that I should build my own server, but now I'm totally convinced that buying a used rackmount is the way to go. The thing is a MONSTER. Dual xeon cpus, 92GB of ECC RAM, 48TB of WD REs in 12 hotswap bays, 120GB SSD, dual PSUs, quad NIC, IPMI for managing it, etc. I can't believe I was thinking about building something that would have cost me twice as much and would have been way outclassed by this thing.
The noise isn't bad, either, surprisingly. I put it in a closet and I can't hear it with the door closed.

I just finished installing ESXI, FreeNas 9.10, and Ubuntu Snappy (for all of my docker containers that will run the server apps). Couldn't be happier!