FreeNAS + ESXI Lab Build Log

[Maelos]

Thank you for the advice and your post on hard drive failures. I had a good chat about drive failures, RAID, and backups with him today sourcing your article.
As for the drives, I will probably return the 8TB and purchase 4-6x 4TB drives for use in a Z2 array. I have two older (3YO) WD Green 4TB drives and one newly shucked White (Red) drive from Best Buy. Having big drives seems nice for the value, but not really what the RAID, backup, and ZFS systems seem to be about/focused on. I will use the calculator to make sure I have enough room to start and a bit to grow into. Do you advise against used, refurbished, pre-owned, etc drives? Is it just not worth the chance or extra testing?

I purchased 2x (3x 4TB WD Red for $365, $9 off combo) for $730 from Newegg.

 

[Chris Moore]

I run a 4 drive RAIDz1 pool inside the same server with my main storage pool and I backup data to that pool on a weekly basis. It gives me another layer to my backup plan. I also use ZFS snapshots so that I can go back and get something from the snapshot if I accidentally delete or overwrite something. Snapshots are great. They have saved me a time or two. But I don't really worry about a drive failure taking my pool out. I have replaced so many drives since I started using FreeNAS, because my first pool was made with second hand drives sourced from eBay... I am using all new drives since then, but I still watch them like a hawk and have tested, cold spares, on standby to drop into my server at the first hint of a problem. I have replaced a drive for a single bad sector, no hesitation.

 

[Maelos]

The build came out a bit costly but this is a long term, educational build. Thank you all again for the guidance. I will post pictures and updates as the equipment arrives. I am excited, even if the wallet burns.

Just a thought, I could use the SATA DOM (included in the chassis purchase, bonus!) as a SLOG. I have read, but mostly glossed over, the SLOG requirements, but if I eventually intend this for use/experimentation with ESXI, this could be useful. On the topic of ESXI, do you need to dedicate an entire vDev to ESXI, or can it be a volume within the vDev? Please forgive me if I am misusing terms. I am reading through the manual/documentation.

 

[Maelos]

I have all the parts except the rails and the undervolting wires for the fans. I do have a question though - what is the part number for the internal drive bays that attach to the power supply exterior. I still would prefer to have an SSD directly attached to the motherboard for ESXI and other VM OS's.

This https://www.ebay.com/itm/132066265139 looks like what I want, but it does not seem to detail compatibility for the SC-846. I can always drill a few holes if needed, but trying to avoid that if I am able. For reference, here is the page for my chassis: https://www.supermicro.com/products/chassis/4U/846/SC846A-R1200B

https://www.amazon.com/Supermicro-M...coding=UTF8&psc=1&refRID=MWVF99CMPBKDPRXD6RDT ?

 

[Chris Moore]

This https://www.ebay.com/itm/132066265139 looks like what I want, but it does not seem to detail compatibility for the SC-846.

That is only part of the bracket. Not the whole thing. It wouldn't work without the other half of the kit. Hang on, I will find it.

 

[Chris Moore]

I can always drill a few holes if needed, but trying to avoid that if I am able.

No holes to drill.... This bracket attaches to the other bracket you already found and this one is the one that attaches to the side of the power supply. It is a little odd, I guess, how they do it. This bracket allows you to mount a single 3.5" drive internally and the other bracket adapts this to allow you to mount two 2.5" drives.
https://www.amazon.com/Supermicro-MCP-220-84601-0N-Inch-Fixed-Drive/dp/B005ITOXM4

 

[Chris Moore]

You can save a couple bucks purchasing the two here:
https://www.neweggbusiness.com/product/product.aspx?item=9b-11-994-023
https://www.neweggbusiness.com/product/product.aspx?item=9b-1wk-0010-00081

 

[Maelos]

Just found the page (4-19) in the manual too with the numbers. Thanks, as always Chris. I assume the 2x 2.5" adapter will fit just as the 1x 2.5" adapter does
Single: https://www.ebay.com/itm/192059105517
Double: https://www.ebay.com/p/26011373403

 

[Chris Moore]

This price is so low, it almost looks like a mistake:
https://www.newegg.com/Product/Product.aspx?Item=N82E16816101828

 

[Chris Moore]

There should be room for two of the brackets. Here:

and here:

 

[Maelos]

I don't know how you find all that so quickly Chris, but I owe you a beer/glass of wine/cider/drink etc. Thank you for saving me some more $. I tried to order two and got an error saying the stock was too low, but ended up at $11.34 with $7.35 shipping. I will have to untangle the nicely corded (the eBay guy who sold me the combo did a really nice job) to find the SATA power pins.

 

[Chris Moore]

I don't know how you find all that so quickly Chris, but I owe you a beer/glass of wine/cider/drink etc. Thank you for saving me some more $.

Part of the reason I can find it is because I know it is there, the hard part is finding the correct search term to get at it because I can't remember all the part numbers. Google does the rest.

PS. You might need Molex to SATA power adapter. I think I did and I apologize for not mentioning it earlier.

 

[Maelos]

Your GoogleFu is strong.

Edit - I think I have a few of those in my part bags. We should be good. That is one of the power cables that is exposed in his nice wiring so I will go with that. I thought about buying a second set of the SSD/HDD cage, or cancelling and trying to buy two of the $4 sets, but I will hold off for now. I though the second set of 2.5" bays could be used for the additional read and write cache's, but I think I recall these drives needing to be on the PCIe lanes...? Not sure, but may not need them anyway with the amount of RAM I have, we shall see. If I ever get to the point where I need either additional synchronous write or read cache, the $ will have been well spent anywho.

Edits continue: Aww heck with it, I ordered a second from the other seller for $14.99 with shipping. If I just bought Newegg out of stock and it is somehow worth the money for me to ship it to someone (who needs it more than I)...let me know. Yay for future proofing...maybe?

More pictures and such coming later. Of note, it appears I only have 1x 4 pin and 1x molex power adapters left. This mobo is a beast. There are so many pins set to things I will have to read up on, power cable attachments, and other doodads. A side thought, it is tempting to remove the high RPM fans and try to find a way to rig my Cougar fans into the case. There is plenty of room for them to fit, I think, it would just be a matter of designing mounts for them. I have 8x 140mm and 6x120mm HDB Cougar fans that I used in my early life crisis case mentioned earlier in this thread. I will be taking videos to compare and contrast the sound levels with each fan setup. More to come...

Adding link for eventual guide - discuss of flash for write and read 46:27 of https://www.youtube.com/watch?v=LtEuNzTpS7s

 

[Maelos]

Shameless near 48h bump again. I turned on the chassis last night as a noise test (no modifications yet) and was surprised. The internal fan noise was not as bad as I thought it would be, but the PSU noise, especially the high pitch, got a comment from my wife. As an aside, I initially powered it on with only 1 PSU plugged but both PSUs inserted, giving off a nasty, loud alarm which may have garnered the comment from my wife. I will have to do some further testing to make sure what she commented on was not that initial alarm. The high pitched "whine" was hard to distinguish. I even thought at one point it could have been coming from crickets/bugs outside. I took some video and will be using it for the guide and comparison later. Wit the fans not being as bad, and the PSU seemingly a bigger problem (pending further tests) I almost wonder if I should return the Noctuas and do some modding to the chassis to fit my old 800W PSU in place of the Supermicro 1U PSUs. Has this been done before, any thoughts? The other option is trying the 920SQ power supply for $95 off of eBay. Does anyone know of a way to silence or reduce the "whine" or high pitch noise from the PSU. Chris, you mentioned dampening on the ventilation side. What exactly did you do?

The other things I am waiting on...

• Internal drive bays for SSDs to be directly attached to the mobo for ESXI storage. I am a little worried I will never be able to power more than two SSDs here as the only remaining power options I have are a molex, a 4 pin CPU (I think), and a 4 pin FAN power supply from the main line. I will have to take a picture of this and post when everything arrives and I get to really building/testing this weekend (hopefully).
• Noise reduction / undervolting wires from China. It was $1.80 with free shipping, but it may be another 1-5 weeks before I get them. Good thing is the fans are less of a worry now.
• Rails - hopefully here today or tomorrow. I'd feel much better having it racked rather than on top of something. Makes it easier to hide aware, possibly muffle, and keep kid safer.

I will continue to do research on the PSU noise and options. I'd rather not buy $200 worth of SQ PSU's, but if is what is best for the long term, so be it. While using a consumer PSU may be quieter initially, losing the redundancy of the PSU and possibly not having the options presented with a chassis/purpose build PSU may not be worth it. I will also be breaking down my previous machine and selling it part by part this weekend as part of this hardware swap. I have been unsuccessful thus far in selling the items as a whole.

 

[Chris Moore]

I almost wonder if I should return the Noctuas and do some modding to the chassis to fit my old 800W PSU in place of the Supermicro 1U PSUs.

I really hope you don't do that. It is easy to damp the high pitched wine by putting some sound damping material between you and the server. Redundant power is one of the big reasons I suggest these chassis.

Has this been done before, any thoughts?

Yes, it has been done. I think it is a bad idea.
https://forums.freenas.org/index.php?threads/lets-talk-rackmount.58012/page-2#post-443469

 

[Maelos]

Thanks Chris. I feel like the child/modder in me got a smack down, but I am here for the wisdom. Its odd, but the 1280 plat SQ PSUs are $25 cheaper a piece than the 920W. I wonder how much less efficient the 1280W would be at my current and future loads and how long it would take to make up the $50 difference in savings...

2x Xeon E5v2 2650 CPUs
16x 8GB DDR3 RDIMM
8-24 HDDs
1-2 SSDs
1 Sata DOM

Hmm, going to see if I can calculate the total wattage used and I may go ahead and order the plat PSUs. Adding https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/ as a link for the guide.

From the S846 Manual these are the compatible power supplies. I have read in other threads, however, that others of the same FF can be used.

R1K28 1280W
MFR Part # PWS-1K28P-SQ
AC Input
1000W Output @ 100-140V, 8-12A, 50-60Hz
1280W Output @ 180-240V, 6-8A, 50-60Hz
DC Output
1000W: +12V/83A; +5Vsb/4A
1280W: +12V/106.7A, +5Vsb/4A

-R1200 1200W (Current PSU)
MFR Part # PWS-1K21P-1R
AC Input
1000W Output @ 100-140V, 11.5-8A, 50-60Hz
1200W Output @ 180-240V, 8-5.5A, 50-60Hz
DC Output
1000W: +12V/83A; +5Vsb/4A
1200W: +12V/100A; +5Vsb/4A
with PDB: +5V: 50 Amp, +3.3V: 30 Amp, -12V: 0.6 Amp

R920 920W
MFR Part # PWS-920P-SQ
AC Input 100-240 V, 50-60 Hz, 11-4.5 Amp
DC Output +12V/75A; +5Vsb/4A

Links below for the guide eventually, research for now.
https://forums.servethehome.com/ind...ws-501p-1r-pws-920p-sq-noise-comparison.8747/
https://www.reddit.com/r/homelab/comments/48mm0b/quieting_supermicro_servers_lots_of_linksinfo/
http://www.avsforum.com/forum/26-ho...ss-expensive-norco-4220-4224-alternative.html Nice just to see how easy it is even if advised against.
https://www.reddit.com/r/homelab/comments/54f4np/anyone_have_a_list_of_supermicro_power_supplies/
https://forums.servethehome.com/index.php?threads/supermicro-noise-levels.1004/
https://arstechnica.com/civis/viewtopic.php?f=7&t=67921 ($loss for too big of a PSU = it may save <$10/year to get the smaller PSU)

Worth $140-180...or just use some towels. https://outervision.com/b/1r53iL so 451W is the estimate with 10 drives and an SSD

 

[Chris Moore]

http://www.avsforum.com/forum/26-ho...ss-expensive-norco-4220-4224-alternative.html Nice just to see how easy it is even if advised against.

It can be done, and it may be quiet, but a big part of getting a rack mounted server (for me) was the redundant power supplies. I don't need to worry about a power supply taking my storage array out. I had a power supply go bad and my server had 3, so I ran from the remaining two while I waited for a replacement to come in. It doesn't happen often, but it is a nice feature that you paid for. Why would you want to rip it out? It is kind of like buying a car with leather seats and ripping them out to use folding chairs instead.

 

[Maelos]

Chris, do you have any experience with the SQ PSUs? How will having excessive capacity for a PSU (1280W) affect the bill? I am waiting on the person who sold me the chassis combo to see if he has any SQs around he wants to sell, otherwise I may buy two after some further testing with my wife.

 

[Chris Moore]

The consumption of electricity is affected by the efficiency of the power supplies because some power is lost in the PSU. I just added 16 more drives to my system and it increased the draw from the UPS from around 220 to around 380, but the supply is rated for 1600 watts, if I recall correctly. There are 32 x 3.5 inch drives in this unit with room for 16 more.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Maelos]

Update 3/12/18

Over the weekend I had a chance to do some further testing, install the active coolers, and I received all parts except for the server rails. What follows are the results of the tests, issues, and solutions. Finally, I have a few questions for those with X9 Supermicro motherboards.

Sound testing

• First, and most importantly, the wife test. After my wife was settled in and my child asleep I had turned on the server and had her give me some more feedback. The great news is that the loud "whine" she heard last time was the alarm from having both PSUs in without both being plugged in. As I won't make that mistake again, this saved me $140 for new PSUs (at least for now). She also stated that the noise level was acceptable, which is also good because that was without any modification and should be the loudest they should be.
• I took a few videos, but the files seem to be too large and the noise cancelling / lack of scientific approach / lack of measurement device makes them less valuable. If anyone would still like to hear/see the footage please just ask.

Replacement of Passive with Active CPU Coolers


Yuck, the paste on the passive coolers was all over the place. I had to take the CPU out (neat little learning experience there, double locks!) and clean it and the socket.


This is a comparison of the old, passive heat sink vs the new, active heat sink (without fans yet). It took me a good few minutes to read and understand the directions of how to switch and fit the base supports for the block to the Narrow ILM socket. The plastic retention/safety circles required brute force via pliers and were a pain to get off and move over. When I applied the fans though, I had some clearance issue initially...


Luckily though, the fans were easy enough to adjust. I ended up having the power cables from the fans come out the top so I could steer clear of the RAM.


The finished product with the loose wires...


Speaking of wiring, I do have the Molex-2x SATA power cable for the ESXI SSD drive(s). I hope that I can use the 64GB SATA-DOM to redundantly install both ESXI and FreeNAS, but I don't want to limit the SSD's 500GB capacity. I will have to see how that goes. I do however have a newly discovered issue. I have 9 fans and 8 fan headers. The good news is that I can use previous experience and equipment from my water cooling build to fix this. My questions are...

• Does anyone know the amperage or total wattage that the X9 series motherboards can provide via their fan headers? I would only need one of these fan headers to run 2 fans and I could make them two redundant(ish) fans.
• Has anyone done a comparison on fan speed and noise when using either software to control or hardware (undervolting)? What are your experiences? I wanted to try the software first, as I could control it remotely with IPMI and without the need to add additional components to the build, but if undervolting with resistant wires works better, I have he parts for that too.

I am excited. The rails come Tuesday :)-/), but I may start with the ESXI install and get familiar with the BIOS(es) tonight.

Edit - the supermicro page for the chassis states 8 fan headers for 8 fans. I will look up the draw on the chassis fans vs the draw on the Noctua fans to get a better idea. Noctua Input Power 0,96 W from https://noctua.at/en/products/cpu-cooler-workstation-server/nh-u9dx-i4/specification and the chassis fans are 0.6 and 0.35 amp for mid and rear fans respectively. I can't recall the formula exactly, but I think the Noctua fans are far under the chassis fans. 12v fed * amp - Watts. So a 1 watt Noctua fan is 0.125 ish amps or a third of a rear fan.