Second FreeNAS build after 9 Years!

[walks]

Hello all! So I moved my introduction info to the "Introductions" forum to avoid unnecessary reading for everyone in this post. With that said, below is the new FreeNAS build I have in progress and hoping to get some additional feedback from the community on:

Motherboard: X10SL7-F($177.02)
Processor: E3-1220 v3(came with board)
Case: CSE-826BE16-R920LPB/SC826E16-R920LPB with BPN-SAS2-826EL1 and 2xPWS-920P-SQ ($145)
Case/Trays: 12xSC93301 3.5 HD Trays ($27.68)
Case/Screws: Supermicro Screw Bag and Labels for 3.5 HD Trays (MCP-410-00005-0N), 100 pcs($4.47)
Case/Cable: 19" SATA Discrete to SFF-8087 36-Pin Reverse Breakout Cable ($12.99)
Memory: 4x8GB Micron MT18KSF1G72AZ-1G6E1ZE/32GB Total ($75.96)
OS Boot Drive: Innodisk SATADOM-ML 3SE Series 64GB ($29.98)
Storage: 6x4TB HMS5C4040BLE640 ($367.94)

Total: $841.04

Hoping I didn't do too bad on the costs. Only waiting on the 826 case to be delivered now!

This new config is probably overkill based on the fact that the microserver served its purpose reliably for years without performance issues.
We currently don't do any transcoding today as all our media files are in a format that all clients can read without transcoding, but if we do
need to go that route in the future it looks like the E3-1220 v3 should be able to handle it based on the hardware recommendation guide and other forum posts I have seen. I plan on creating 1xRAIDZ2 vdev with the first six disks, and then add another six in a second RAIDZ2 vdev when the finances permit.

Before I finally decided to go this route I was looking at a Denverton setup with consumer hardware, and never thought of leveraging used enterprise hardware before since I assumed the power consumption would be astronomical! But reading up on posts in these forums has convinced me otherwise, and my new setup actually costs me less than the consumer hardware I was considering, although the Denverton represented a big chunk of that cost. And I love having IPMI!!! Performed the bios update, the 2308 it mode update, and ipmi firmware update
using IPMI! We have these capabilities at work, but never thought about the convenience of also having this at home since I never really considered running enterprise hardware at home before due to power consumption concerns.

Now I am trying to get a handle on what the total idle power consumption will be. Below are my estimates:
X10SL7-F: (could not find specific numbers for the board itself)
E3-1220 v3: 20 watts total
MT18KSF1G72AZ-1G6E1ZE: 4x2watts 8 watts total??
HMS5C4040BLE640: 6x4.9 watts 29.4 watts total
2308 Controller: 8 watts total
SATA-DOM: 1 watt total?
PSU: ??
FANS: 10 watts total??

I have been searching the net/forums to gather the info but coming up short on some of the numbers. So I am coming up with roughly idle 76.4 watts with just the numbers above but not sure how accurate that is so please correct me where ever I am wrong. I know this will may consume more than my current Microserver setup, but hopefully not much more.

Also, saw a post about the ATX 24pin and 12v 8pin connectors from the power distributor possibly being too short? So wondering if I may need
to get extenders for those. I know in the post by mattbbpl(https://www.ixsystems.com/community/threads/build-double-check-painting-by-numbers.34932/) post #33 he shows a picture of the board installed and the connectors not reaching. He states he has the CSE-826TQ-R800LPB with the 800watt power supplies which uses the PDB-PT825-8824 power distributor, whereas the case I purchased with the 920watt power supplies has the PDB-PT826-8824 power distributor. Is it possible that the connector lengths are different depending on the PD you get? Have not seen any other posts in regards to the connectors being too short when installing the X10SL7-F in a CSE-826 case so any feedback from anyone on this topic would be greatly appreciated as well!

 

[ethereal]

https://www.ixsystems.com/community/threads/proper-power-supply-sizing-guidance.38811/

 

[walks]

https://www.ixsystems.com/community/threads/proper-power-supply-sizing-guidance.38811/

Thanks for that ethereal! Have not seen that guide yet. The the one example that is closest to my setup is:

2) For an E3-1230v3 (32-98W board+CPU, 12W memory):

• 5-6 Drives: 360W peak, 132W idle -> SeaSonic G-450

But I was really hoping to save more. That would put me at roughly $13 per month based on those rough pre-calculated numbers!

So below are the rough numbers I have come up with based on that guide and power info I have scrounged up on my specific hardware:

 

[walks]

So just received the case and can now confirm that the ATX 24-pin and 12V 8-pin power appear to be long enough without extenders on this particular chassis. I was also pleasantly surprised that the case came with the 2x2.5 HotSwap bays on the back. Here are some shots of the case before I start putting everything together:

Part/Model Info:

So this particular part number indicates that the case originally came with the X9DRD-7N4F-JBOD motherboard which was 12" x 13" in size. And from some notes they left on the case was running dual Xeon E5-2660 V2 processors.

The photo below with the large black piece appears to be another type of air shroud? Although I am used to seeing the all clear air shroud that completely encloses the area its installed on in other photos where an air shroud is present. I am assuming if this was used with the larger X9 board referenced above, it should hopefully be more than effective helping to cool my X10SL7-F with E3-1220 V3 processor with passive heatsink? I will have to see if the shape of this shroud will line up correctly with my board:

Another photo without the air shroud:

Here is a shot of the unexpected bonus 2x2.5 rear hot swap bays. Although I did order a satadom already, I am wondering what other freenas users end up using these for?

And here is a shot of the dual PWS-920P-SQ power supplies:

 

[Snow]

I love Supermicro stuff It just fits and works. Love the pics Hopefully we get to see a finished product. I feel like they build there stuff to be the Swiss army knifes of servers.

EDIT:On a plus note the little extra 2x2.5 bay cost like $100 to get on the side so it is a plus for a boot drive or 2x jail ssd's.

 

[CraigD]

Now I am trying to get a handle on what the total idle power consumption will be

Mine is 230w idle, this was obtained from a a kill-a-watt type device

X10SL7-F
E3-1271 v3
Noctua NH-D15 CPU Cooler (2x 140mm Fans)
32GB RAM
M1015
2x SAS Expanders
7x 5 in 3 hot swap bays (each has a 120mm Fan)
4 in 1 2.5" hotswap bay (includes 40mm Fan)
2x 120mm Fans

7x 8TB Ironwolf drives
1x 8TB WD Red drive
9x 4TB WD Red drives
8x 3TB WD Red drives

2x 500GB crucial mx500 SSDs
120GB SSD (Boot)

My guess your idle draw will be around 100-150w idle

Hopes this helps

 

[Snow]

I would say peak is around 500-600W for your system CraigD ?
My primary nas only peaks at 400w that with 7 SSD and 18-3/4TB-Hdds
How hot do the 8Tb Disks run ?

 

[walks]

I like the Swiss Army Knife analogy and love that it just works! I mean this is some really old/used hardware we are talking about and everything just works!

@CraigD - yes, I think I am going to fall into that 100-150 ballpark you mentioned unfortunately :(
I was actually hoping to get the E3-1271 v3 or E3-1270 v3 like you have because I believe they both have almost the identical idle power consumption as the E3-1220 V3 that I have. But when I came across the deal on the board+cpu I decided I could live with the 1220 for now as long as it was not consuming more power at idle.

 

[walks]

@Snow - I see in your builds you are running the X8 and X9 boards. From what I have read so far, it seems that major power saving gains came with the X9 and newer boards. What is your idle power usage like?

 

[Snow]

100-200 watts on the x9, The X8 is 305-375 Most you gain from the X10 is around 5-7%. X11 Is 15% so for what you pay, you do not gain very much! I much rather stick the extra money in to more ram, hdd's or a faster Cpu. 15% is not very Much in performer and the power is close to the same as the X9 I Run at around 250-370 Watts a hour but that is a 24bay, Dual Cpu system with a crap ton of ram Also my system is really never Idling. If you have not picked up the mother board look at the Dual Xeon setup with a good amount of ram. It well last you another 10 years. I plan on getting a X10/X11 In 2 or 3 Years to replace the X9 for my main system as the cost for ddr4, broadwell and skylake gen Xeons come down in price.

 

[walks]

I agree with you that the power consumption differences from X9-X11 are not enough to warrant paying a premium for one over the other. Which is why my main focus was to find the the best deal with the features I wanted in the X9-X11 range and ended up purchasing the X10SL7-F board which came with the E3-1220 V3. So I am working with that for now.

 

[walks]

This is going a little slower than expected because now I have a screw issue. The standoffs for the motherboard are there, and I moved them around to match up with the holes on the X10SL7-F. I then assumed that I could just use some of the many motherboard screws I have laying around, but apparently not. Finding actual Supermicro motherboard screws seems to be near impossible, so I found some on ebay that seemed to match up with Ericloewe's post where he describes the screws in post#5 (https://www.ixsystems.com/community...4-203b-screws-for-mainboard.50194/post-346387)
Things to pay attention to:

• Standoff thread: 6-32 or M3
• Screw thread: 6-32 or M3
• Standoff length

Based on the above, I am assuming the screw thread length is probably .25" or 5mm.
I ended up purchasing these off ebay:
https://www.ebay.com/itm/6-32-Pan-Head-Computer-Screws-Hard-Drive-PC-Case-Standoffs-25-to-500-Closing-Out/352645578450?ssPageName=STRK:MEBIDX:IT&var=621990110351&_trksid=p2057872.m2749.l2648

 

[walks]

Well for now, while I am waiting on the screws, I positioned the board in the case, and then added the air shroud to see how it would fit:

And here is another shot of it from the rear of the case:

That middle support (circled in red) is actually resting on the heatsink.

Below are pictures of it tilted up with the part of the support that touches circled in red:

Should I be concerned that this is touching the heatsink?

 

[CraigD]

I would say peak is around 500-600W for your system CraigD ?
My primary nas only peaks at 400w that with 7 SSD and 16-3/4TB-Hdds
How hot do the 8Tb Disks run ?

I'm idle 99% of the time, I dont think I've seen 400W

They run hot I hit 45C twice this year, idle temps below

Remember the 8TB Ironwolf drives are 7200 RPM all the other drives are 5400 RPM

Hope this was helpful

Code:

=== DRIVES ===
   da0:   28C [3.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD30EFRX-68EUZN0)
   da3:   28C [3.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD30EFRX-68EUZN0)
   da4:   28C [3.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD30EFRX-68EUZN0)
   da5:   36C [8.00TB] XXXXXXXXXXXXXXX      Seagate IronWolf (ST8000VN0022-2EL112)
   da6:   28C [3.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD30EFRX-68EUZN0)
   da7:   36C [8.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD80EFAX-68KNBN0)
   da8:   36C [8.00TB] XXXXXXXXXXXXXXX      Seagate IronWolf (ST8000VN0022-2EL112)
   da9:   35C [8.00TB] XXXXXXXXXXXXXXX      Seagate IronWolf (ST8000VN0022-2EL112)
  da10:   34C [8.00TB] XXXXXXXXXXXXXXX      Seagate IronWolf (ST8000VN0022-2EL112)
  da11:   34C [8.00TB] XXXXXXXXXXXXXXX      Seagate IronWolf (ST8000VN0022-2EL112)
  da12:   33C [8.00TB] XXXXXXXXXXXXXXX      Seagate IronWolf (ST8000VN0022-2EL112)
  da13:   28C [3.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD30EFRX-68N32N0)
  da14:   28C [3.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD30EFRX-68EUZN0)
  da15:   28C [3.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD30EFRX-68EUZN0)
  da16:   27C [4.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD40EFRX-68N32N0)
  da17:   26C [4.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD40EFRX-68N32N0)
  da18:   27C [4.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD40EFRX-68N32N0)
  da19:   28C [4.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD40EFRX-68N32N0)
  da20:   28C [4.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD40EFRX-68N32N0)
  da21:   27C [4.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD40EFRX-68N32N0)
  da22:   26C [4.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD40EFRX-68N32N0)
  da23:   27C [4.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD40EFRX-68N32N0)
  da24:   25C [4.00TB] XXXXXXXXXXXXXXX      Seagate IronWolf (ST4000VN008-2DR166)

  ada1:   36C [8.00TB] XXXXXXXXXXXXXXX      Seagate IronWolf (ST8000VN0022-2EL112)
  ada2:   35C [3.00TB] XXXXXXXXXXXXXXX      Western Digital Red (WDC WD30EFRX-68EUZN0)

 

[Snow]

I am pretty sure they make a couple different shrouds. MCP-310-29001-0N Link If you want to spend then extra 50$ on it.

Supermicro Parts List and other goodies

 

[walks]

I am pretty sure they make a couple different shrouds. MCP-310-29001-0N Link If you want to spend then extra 50$ on it.
View attachment 30882
Supermicro Parts List and other goodies

Looks nice, but I am good with the one I got lol. Would not want to pay that premium unless it was absolutely necessary. The part number on the one I got was MCP-310-29014-0N by the way, and although I am still somewhat concerned about it touching the heatsink, the processor tends to hover around 35-38 degrees so far, even during the burn in tests, so heat potentially melting it does not appear to be an issue.

And by the way, I am kind of shocked how quiet this is so far. It is of course loud when it first powers up and then goes down to a low hum once running with the 3 fans hovering around 2600-2800 RPM's. I still need to implement the script to control fan speed and follow Ericloewe's guide for setting the thresholds, but trying to hunt down the full manufacturer's specs for the FAN-0126L4 fans first. I am now very interested in seeing how much how much quieter the system will get after implementing the threshold's and the script.

 

[walks]

Ok, correction on my part about the heat. So when I started the burn-in tests cpu was 35-38 as previously stated, but that was during the short/long smart tests. When the check for bad blocks started, cpu temps got up to 50 degrees and drives got as high as 36 degrees.

I did get the spinscripts setup though, and that appears to be dealing with the temps accordingly.

 

[Snow]

on sale for 11$, Never mind thats the wrong one. Keep a eye out on ebay yours will pop up then pick it up. Unless you want to get a heatsink with the built in fan for like 35-40$ I would just find the shroud on ebay or pay the prem on amazon. Ive seen temps up to 50-65c not that it is bad but not that it is good! less heat=longer life time.

 

[walks]

I love Supermicro stuff It just fits and works. Love the pics Hopefully we get to see a finished product. I feel like they build there stuff to be the Swiss army knifes of servers.

EDIT:On a plus note the little extra 2x2.5 bay cost like $100 to get on the side so it is a plus for a boot drive or 2x jail ssd's.

So after Snow's comment on the 2x2.5 bay he inspired me to source a pair of 256GB SSD's, and install them in the bay's to host my jails:

Paid $23 for each 256GB SSD and $4 for each 2.5" tray, $54 total shipped.

Here are some other updates to my build:

Sourced a " Innodisk SATADOM-ML 3SE Series 64GB" for the boot drive. I know 64GB is overkill for the boot drive but for $30 bucks, why not:

Had to perform some surgery on the connector as this came with a 4 position connector for power on Dell servers.


I ended up cutting the existing pico blades off of the wire and then soldered them to pins on a connector I sourced from a old case fan. I then hooked it up to one of the fan power leads coming from the power supply:


**A note on the SATA-DOM. Not knowing much about these before getting into this new build, I assumed that I should also perform tests on it as well prior to putting it into production. However, the tests kept failing, and the smartmon tests would just immediately error out leading me to believe I had a bad SATA-DOM. But found a post from @cyberjock "https://www.ixsystems.com/community/threads/freenas-boot-smart-testing.35927/post-224116" where he shares that although the sata-dom may be smart capable, it does not necessarily mean that it supports smart testing. So after discovering this, I made sure smart monitoring was enabled and moved on.

Replaced the solid i/o shield with a grated i/o shield that should hopefully place less restriction on airflow:

 

[Snow]

Looking good you will love the speed on the jails, best thing I ever did made plex really snappy to!