Mr Rackable Supermicro build questions

[pcmofo]

After going back and forth with various issues with my non-ecc consumer-grade FreeNAS build I looked at moving on to a Supermicro case/motherboard upgrade that would meet my needs for the next 5+ years. I'd transfer my existing 8x 2tb pool and replace/expand from there in the future.

I saw a few people here go with used supermicro systems from MrRackable on ebay and I'm trying to find the right system at a good price point for what I need my NAS to do.

The NAS will be rack mounted in a separate closet off of the kitchen so it can be noisy but I would prefer SQ power supplies.

I use AFP shares to edit photos in Lightroom and video in Premiere over the gigabit LAN. Currently I get around 100Mb/s and I'd like to keep or increase that speed.

I have 2 Amazon Fire TV boxes running Plex clients as well as other TV's and computers/phones that run Plex clients. The current NAS runs Plex server in a jail.

I figure I can go one of three directions on this build...

1) I buy just a case/psu combo from Mr Rackable and fill it with newer supermicro motherboard and processor etc.
2) I buy a 12-bay 2u case (this one seems loaded) and eventually move to 2x RAIDz2 vdevs of 6x drives each and keep swapping out/upgrading the drives every 3-5 years with larger ones

3) I buy a 36-bay 4u case (like this one) which comes with the QC quiet PSUs and is way more than I need today space wise, and I eventually fill it

I'd like to spend no more than $1,000 if possible. Again, I would need everything but the drives.

Does anyone have an opinion on what which solution might work best for me given my usage? I am not extremely familiar with Xeon architecture or supermicro hardware to know what to look for, eg if I am buying a setup with an incompatible SAS expander, or if one Xeon would better suit my needs than another. I am open to suggestions.

 

[Fuganater]

I bought a SC846-E1 from the same vendor for just over $300 shipped. Came with dual PSU and rails, no system. I added my own system because I wanted the newest generation of Supermicro boards (X10), not something 6+ years old for many reasons.

 

[pcmofo]

I bought a SC846-E1 from the same vendor for just over $300 shipped. Came with dual PSU and rails, no system. I added my own system because I wanted the newest generation of Supermicro boards (X10), not something 6+ years old for many reasons.

Cool. How is the noise level on that case?

 

[tvsjr]

Be careful, the MrRackables systems that I looked at have the SAS1 expander. That limits you to ~2TB drives. I found another seller (now gone) that had chassis with the SAS2 backplane.

As far as noise, they're rack mount servers, and they require fans with high static pressure to get enough airflow across all the drives and through the backplane. Quiet they ain't.

 

[Fuganater]

Cool. How is the noise level on that case?

Loud enough not to sit in a living space. My office is next to the living room and when the door closed you can still hear it a bit. All the drives stay around 30C.

Be careful, the MrRackables systems that I looked at have the SAS1 expander. That limits you to ~2TB drives. I found another seller (now gone) that had chassis with the SAS2 backplane.

As far as noise, they're rack mount servers, and they require fans with high static pressure to get enough airflow across all the drives and through the backplane. Quiet they ain't.

Not all of their producst are SAS1 expanders but lots of them are. I am only using 2TB drives so that is not an isue. Thereare several sellers that have the 846 chassis with the SAS2 expander and they start around $450. I was told on here people have snaggd them as low as $300.

 

[Fuganater]

Here is one with the SAS2 backplane. You could probably get him down to $400 shipped.
Here is one with the SAS1 backplane. The price is pretty awesome but it does not come with rails. I've seen rails for around $60-$75.

You just need to keep an eye out and watch daily for a good deal.

 

[pcmofo]

Here is one with the SAS2 backplane. You could probably get him down to $400 shipped.
Here is one with the SAS1 backplane. The price is pretty awesome but it does not come with rails. I've seen rails for around $60-$75.

You just need to keep an eye out and watch daily for a good deal.

Awesome. Thanks for the links. I am thinking the SC846E16 style case with SQ power supplies and putting in more modern hardware might be the best route. Getting a SAS2 back plane with the single SFF connector seems like a good choice.

 

[pcmofo]

I ended up ordering the following parts through a few different eBay sellers.
Case: SC846E16-R1200B
Power Supply: PWS-920P-SQ
Backplane: BPN-SAS2-846EL1
RAID Card: IBM M1015

Now I just need to pick a modern 2011 compatible SM motherboard, processor, and ram and I should be in business. Any suggestions to stay under ~$800 with a dual processor board (but only a single processor for now) and 32gb ecc ram?

 

[tvsjr]

I bought a complete server with the 847 case (36 bay) and it came with the X9DRi-LN4F+ motherboard. That gives you two E5/E5v2 processors, LOTS of slots for RAM, and 4 gig interfaces (I've got two in LACP for the Samba stuff and management, and the other two in LACP for NFS to VMware).

That motherboard is available on ebay in the $300 range.

For processors, I ordered two from here: http://www.ebay.com/itm/281832966856?_trksid=p2057872.m2749.l2649&ssPageName=STRK:MEBIDX:IT
Mine work fine after extensive burn-in testing and are NOT engineering samples as best I can tell (and as identified by the Intel processor tool). I paid $150/ea. (make an offer)

That gives you $200 left over for RAM. I bought 4 16GB sticks, Samsung models that were on the approved Supermicro list, for $280.

As always, if you're buying from eBay, make sure you conduct good burn-in testing!

 

[pcmofo]

I bought a complete server with the 847 case (36 bay) and it came with the X9DRi-LN4F+ motherboard. That gives you two E5/E5v2 processors, LOTS of slots for RAM, and 4 gig interfaces (I've got two in LACP for the Samba stuff and management, and the other two in LACP for NFS to VMware).

That motherboard is available on ebay in the $300 range.

For processors, I ordered two from here: http://www.ebay.com/itm/281832966856?_trksid=p2057872.m2749.l2649&ssPageName=STRK:MEBIDX:IT
Mine work fine after extensive burn-in testing and are NOT engineering samples as best I can tell (and as identified by the Intel processor tool). I paid $150/ea. (make an offer)

That gives you $200 left over for RAM. I bought 4 16GB sticks, Samsung models that were on the approved Supermicro list, for $280.

As always, if you're buying from eBay, make sure you conduct good burn-in testing!

That's a good chunk of horsepower you have there! Can you recommend anything thats available via Amazon or Newegg?

 

[tvsjr]

You're going to pay a lot more for new stuff. If you aren't stuck on staying in the LGA2011 family, I would seriously consider the new X11 boards for the E3 v5 processors. That lets you get to 64GB RAM total. Some minor case modifications may be needed (standoffs) but nothing too onerous, I don't believe.

 

[nightshade00013]

Here is one with the SAS2 backplane. You could probably get him down to $400 shipped.

The one problem I see with this is that you are still probably going to have to buy ram and cpu's as well as a sas controller. The X8DTE-F board does not have the built in SAS controller. That is if you intend to actually use the included board.

 

[pcmofo]

I bought a complete server with the 847 case (36 bay) and it came with the X9DRi-LN4F+ motherboard. That gives you two E5/E5v2 processors, LOTS of slots for RAM, and 4 gig interfaces (I've got two in LACP for the Samba stuff and management, and the other two in LACP for NFS to VMware).

That motherboard is available on ebay in the $300 range.

For processors, I ordered two from here: http://www.ebay.com/itm/281832966856?_trksid=p2057872.m2749.l2649&ssPageName=STRK:MEBIDX:IT
Mine work fine after extensive burn-in testing and are NOT engineering samples as best I can tell (and as identified by the Intel processor tool). I paid $150/ea. (make an offer)

That gives you $200 left over for RAM. I bought 4 16GB sticks, Samsung models that were on the approved Supermicro list, for $280.

As always, if you're buying from eBay, make sure you conduct good burn-in testing!

I put in offers on two of the processors you linked to, and on the motherboard. I looked at the memory that is compatible but it appears that 16gb sticks are only available in ECC registered and are still $100 or more each.

I'd love to know what ram you bought and what processor heat sink you are using. As best I can tell from the PDF I need two SNK-P0048P (passive) or SNK-P0048AP4 (active)

It also looks like their might be a built in SAS expander on the motherboard itself. I'm guessing I will still want to use the M1015 though?

And while I was typing this up my offer was accepted on the 2x processors.woohoo!

Looks like I am a motherboard, heat sink, and SAS cable away from a complete server!

 

[pcmofo]

My motherboard offer was accepted and I picked up the X9DRi-LN4F+ for $234 used. Ordered a Monoprice 1m Mini SAS 36-Pin SFF-8087 Male to Mini SAS 36-Pin SFF-8087 Male Cable. ($10)
All thats left is RAM and heat sinks, both of which I could use some advice on.

 

[tvsjr]

I'm running the SNK-P0048AP4 active heatsinks. I had the passive versions installed previously and they worked, but the processors would get hotter than I was comfortable with under full load. Then again, I'm picky. The extra fans also seem to prevent the system fans (which are high static pressure 80x38 7,000 RPM screamers) from winding up in response to increased CPU load. Sitting in an air-conditioned home at comfortable room temperatures, I have to make an effort to load the system to a point where the fans will spin up above their normal "optimal" speed (around 3500RPM). This improves the WAF (Wife Acceptance Factor - a critical value in my builds).

The specific board I recommended has lots of SATA on-board, but no SAS. I have a 9211-8i (functionally equivalent to the M1015 - don't forget to flash to IT mode!) with one cable to the front backplane (24-bay) and one to the rear (12-bay).

This is the RAM I bought - http://www.ebay.com/itm/281833201158?_trksid=p2060353.m2749.l2649&ssPageName=STRK:MEBIDX:IT - I offered $280 for 4 sticks.
Of course, after buying it, I had another 256GB appear - so I threw an additional 64 in my NAS, and the remainder will go in my two new VM hosts once I build them. Grab the approved model numbers from the Supermicro hardware list and start plugging them into eBay, see what pops up. Here's an example: http://www.ebay.com/sch/i.html?_from=R40&_sacat=0&_nkw=HMT42GR7MFR4C-PB&_sop=15

I would recommend sticking with registered ECC memory on this board. If you stick with 16GB DIMMs and find yourself wanting for more cache, you can get to 384GB without going to ludicrously expensive 32GB sticks. This also gives you lots of memory to run multiple jails, should you so desire. Personally, I'm letting my NAS be a NAS... all the other stuff will run on the two ESXi hosts.

Once you get everything together, make sure you allocate a week or two to testing. You need to heavily test all of the components (CPU, RAM, drives) to ensure stability.

 

[Fuganater]

The one problem I see with this is that you are still probably going to have to buy ram and cpu's as well as a sas controller. The X8DTE-F board does not have the built in SAS controller. That is if you intend to actually use the included board.

I would resell any X8 board. They are not worth the power they consume anymore.

Personally I am not a fan of passive heat sinks so I only buy active ones. I got Supermicro's 4U active cooler for all my boards.

I picked up that same board from Hardforums. I am putting 2x E5-2670 in it.

Sent from my SM-T700 using Tapatalk

 

[pcmofo]

I'm running the SNK-P0048AP4 active heatsinks. I had the passive versions installed previously and they worked, but the processors would get
This is the RAM I bought - http://www.ebay.com/itm/281833201158?_trksid=p2060353.m2749.l2649&ssPageName=STRK:MEBIDX:IT - I offered $280 for 4 sticks.
Of course, after buying it, I had another 256GB appear - so I threw an additional 64 in my NAS, and the remainder will go in my two new VM hosts once I build them. Grab the approved model numbers from the Supermicro hardware list and start plugging them into eBay, see what pops up. Here's an example: http://www.ebay.com/sch/i.html?_from=R40&_sacat=0&_nkw=HMT42GR7MFR4C-PB&_sop=15

Once you get everything together, make sure you allocate a week or two to testing. You need to heavily test all of the components (CPU, RAM, drives) to ensure stability.

Thanks for all the great info! I ordered two of the active heat sinks, I'm hoping those along with the SQ power supplies will quiet down the system enough to live in a closet off the kitchen. I may also replace the internal case fans if they are noisy or the non PWM type.

I put in some offers for ram, its going for around $85/16gb.

My plan is to leave my current NAS running as-is, follow the burn in and testing guide and let the server burn in for a few weeks to make sure their is no issues with the 95% used ebay-surced components. And yes, I have to remember to flash the IBM card to IT mode. From there I will toss an old drive or two in the box to make sure FreeNAS works and can see all the drive bays. If everything checks out I will move over my config and existing drives to the new box.

Hopefully this new build will add stability and performance to my NAS setup. I'm also hoping to squeak another 6 months out of my drives (8x 2tb Seagate green) until I can replace them with 6x 4tb WD Red or Red Pro drives in a new zpool. Then the current drives will be used as backup in their own pool until they die.

 

[tvsjr]

The case fans (at least the stock ones) are PWM. However, they are so noisy because they are high static pressure... they have to be able to draw air across the hard drives through the holes in the backplane, which aren't very large. If you do swap them, you need to pay very close attention to your HDD temps (and not just one drive - check them all - I find the drives in the top bays run 4-5C hotter than those in the bottom bays) to ensure you are getting sufficient airflow.

Once the remodel is complete in our new home, I will have a dedicated server closet with its own HVAC... I'm hoping that if I can keep the closet cooler, then I can swap to slower/quieter fans. However, in a 72 degree house with the existing fans, I'm still seeing drive temps in the 42C range, which is on the high end of acceptable for me.

 

[pcmofo]

.

The Supermicro case was delivered but I noticed that none of the drive bays came with screws or the plastic dummy inserts. I ordered screws online but I can't find the dummy inserts. I am assuming that having 8 hard drives in 8/24 bays and nothing in the other bays will effect the air flow and cooling of the 8 drives vs all bays being full with either a drive or the dummy. I can't seem to find much information about this though.

All of the critical parts have arrived, with the exception of the RAM. So it looks like Monday I will be able to boot the system. A test boot shows everything is hooked up correctly to the case and PSU, the PSU works, as do all the fans. I of course get the error beeps for "no memory found". So far the fan's don't appear to be that loud but it's still too soon to tell. For whatever reason the case I ordered came with a X8DTE-F motherboard pre-installed... which I promptly removed. It also did not ship with the correct rails, so I'll definitely be contacting the seller about that, I'm just waiting to see if everything posts correctly when the ram gets here.

 

[nightshade00013]

The 1U I got didn't come with screws or dummies. I had a bunch of laptop screws I had scavenged and just used those. For the cooling as long as you have fans right behind the drives they will draw air through. The 1U case had fans on only one side and during badblocks on the two drives installed the temp only got up to 26c. More fans are being added as once four drives are in place it will need a couple extra's to draw over t

Actually have a set of rails that came with mine sitting here. Let me know and we can work something out.