FreeNAS + ESXI Lab Build Log

[Chris Moore]

So it looks like 8+ with room for 4 more VMs as needed. That being said, FreeNAS would likely dominate the need for resources. This is also a preliminary list and could grow or shrink as I learn.

You should have no problem running two VMs for each physical core but your 'bottleneck' will be disk IO. That is where having cache drives comes in to play. This is also the advantage of a board with many PCIe slots to add things like the PCIe NVMe cards.

My downstairs closet is across from the AC/Heat. If I was to go with this board and its server grade fans/PSUs, is it possible that I can better soundproof the room to reduce the noise?
Also, I am just now reading about vibration being a problem for non-enterprise disks. Will this be the case for non NAS drives, or even non-pro NAS drives such as the shucked White/Red WD drives I plan to use.

The amount of sound generated by the 24bay system (I have one of those chassis) can be mitigated by running the fans at a lower speed. This will still keep the drives cool, but you have to change the heat sink on the CPUs to active models. These systems usually come with passive CPU coolers that work fine with high airflow but don't cool adequately with low airflow. I tried it. Lesson learned the hard way. Once you slow the chassis fans down and put active heat sink on the CPUs, I had mine in my office. It does take a little noise baffle on the exhaust side because the power supply fans make a high pitched whine, but it is not too bad.

PS. I have not had any problems that I would attribute to vibration and I don't use NAS drives in my servers at home.

 

[Maelos]

To confirm before purchasing, does https://www.ebay.com/itm/Supermicro...708722?hash=item56a3590932:g:2tsAAOSwpvZZ5iIc work for the 846? According to the SuperMicro website, MCP-290-00014-00 (https://www.supermicro.com/products/accessories/index.cfm?Type=17), is what is required.

 

[Chris Moore]

Ohh, I found that the 846 chassis has space for internal non-hotswap drives, including 2.5

Yes, that is how I have my boot drives mounted.

It is a nice feature.

I carry over some of the fans from my early build and do a bit of modding (may be possible?)

I have seen some people do it, but I don't suggest it. All I did was slow the fans down with an inline "low noise" adapter.
https://www.ebay.com/itm/5-Pcs-3-Pins-Noise-Reduction-Cable-Lead-for-PC-Cooling-Fan-/182948208544
or if it has 4pin fans
https://www.ebay.com/itm/232107503676
Here is what it looks like empty:

I bought one of these systems a back in 2016 and it came with the X7DBE system board 2 of the Xeon E5450 CPUs and 32GB of (if I recall) DDR2 memory. I have stripped that board out and made some changes, but nothing structural.
The fans in mine were 92mm on the fan wall and 80mm on the rear exhaust. I completely removed the ones at the rear and the plastic air guide because once I slowed the fans, I needed to switch to an active CPU cooler.

 

[Maelos]

interesting. Any thoughts on the rails?

 

[Chris Moore]

interesting. Any thoughts on the rails?

According to the part numbers, this is the correct rail set:
https://www.ebay.com/itm/SuperMicro-MCP-290-00058-0N-2U-3U-4U-5U-Snap-In-Rail-Set/162775862144

 

[Maelos]

Ahh, my mistake. Interesting that the ebay price as actually close to Newegg. It irks me a little that the seller of the box pointed me to, potentially, the wrong set.

 

[Chris Moore]

Ahh, my mistake. Interesting that the ebay price as actually close to Newegg. It irks me a little that the seller of the box pointed me to, potentially, the wrong set.

It is possible that the other set would work, but I don't have any way of knowing. I just went by the part number of the chassis and looked up the compatible rail kit.

 

[Maelos]

I did that, but wanted to see if I was missing something. I'd rather just have it work, so model number it is.

So far the eBay cart is the Chassis Combo, Speed Reduces, and rails. Is there anything you suggest I not buy from Ebay?

• Chassis, Mobo, CPUs, RAM, PSUs - https://www.ebay.com/itm/232656106862
• Rails - https://www.ebay.com/itm/162775862144
  
• Fan noise reducers, https://www.ebay.com/itm/182948208544
  
• UPS, $149 https://www.newegg.com/Product/Product.aspx?item=N82E16842301561
• SAS-SAS $9,
• More drives - May go all out with 5x 8TB WD NAS drives, or may go for something smaller. Heaven knows I may never use that much room but future proofing?

I think I found the same deal but $300 cheaper, https://www.ebay.com/itm/Supermicro-4U-X9DRI-F-2x-Xeon-E5-2650-v2-2-6ghz-8-Core128gb-24-Bay-Server-/112838486677?_trksid=p2141725.m3641.l6368. a
I found that here: https://www.discsandersdiscountstore.com/disc/4u-server.html&price=disc (page of links)

 

[Chris Moore]

You are probably going to want to get tower CPU coolers but they need the narrow mount, if I recall correctly.
I bought Noctua, but I can't recall the model number.

The fan-wall fans of my system were connected to the drive backplane.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[tvsjr]

You'll be fine with red drives. Your bigger concern should be climate control... you're going to be dissipating a fair amount of heat. Doing so into a sealed room like a closet is a bad idea, as the room will heat up substantially. I'm actually working on this problem myself.

 

[Maelos]

You'll be fine with red drives. Your bigger concern should be climate control... you're going to be dissipating a fair amount of heat. Doing so into a sealed room like a closet is a bad idea, as the room will heat up substantially. I'm actually working on this problem myself.

The good news is that the closet has an air vent, is on the bottom floor, and has a wall to outside and a wall to the garage. Depending on how it goes I will make modifications. I think that will be part of the fun, I hope.

Other items I found on the website mentioned above include top loaded Chenbro, Norco, and other well priced Super Micro. I was not able to find anything that can beat that $700 deal, though I am waiting for them to answer a question about the RAM and drive caddies.

 

[tvsjr]

Just remember, it's not about air... it's about air *flow*. The heated air has to get back to the A/C return. If you seal the room up tight, you'll simply increase the static pressure of the room to the point that no air will move, and it'll be just about as bad as having no vent at all.

And yes, I've been doing a lot of thinking and studying on my closet, dealing with the same issues...q

 

[Maelos]

Yeeeee! Order total 1021.64 for the chassis combo, UPS, speed reducers, and rails. The seller replied and answered all of my questions. Here is to hoping I did not just step in a big pile!

The final, purchased build so far is

Supermicro 4U Server, $800 with shipping (https://www.ebay.com/itm/112838486677)

• CSE-846A-R1200B Chassis
• X9DRI-F Motherboard (Supports V2 CPUs / 16 Memory Slots)
• 2x E5-2650 v2 2.6ghz 8-Core 8.0 GT/s / 20mb Smart Cache CPUs
• 16x 8gb PC3-10600R Server Memory
• LSI 9210-8i JBOD Controller
• SAS2-846EL1 Backplane
  
• 24 x Drive Bays
• 1 x 64GB Supermicro SATA DOM (SuperDOM) Boot Drive
• 2x 1200w Gold PSU
• Per message includes all internal cables needed (mini SAS to mini SAS)

3pin Noise Reduction Lead for PC Cooling Fan, $1.80 (https://www.ebay.com/itm/182948208544)
APC 1500 VA UPS, $150 (https://www.ebay.com/itm/382387133959)
SuperMicro MCP-290-00058-0N 2U/3U/4U/5U Snap-In Rail Set, $70 (https://www.ebay.com/itm/162775862144)

Still need drives, active cooling for CPUs, sound proofing for room?

 

[Stux]

FreeNAS throws a spanner in the works when it comes to the Moar Cores or More Mhz debate, mainly becuase SMB is single-threaded (in the current implementation at least).

You are probably going to want to get tower CPU coolers but they need the narrow mount, if I recall correctly.
I bought Noctua, but I can't recall the model number.

The fan-wall fans of my system were connected to the drive backplane.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

This is the Noctua cooler for 4U LGA2011 (etc) installs
https://noctua.at/en/products/cpu-cooler-workstation-server/nh-u9dx-i4

2.6ghz v2 Octacores sounds like a good middle-of-the-road between the gimped low end CPUs Intel makes and the fast single-thread specialists.

https://ark.intel.com/products/75269/Intel-Xeon-Processor-E5-2650-v2-20M-Cache-2_60-GHz

Remember, they turbo to 3.4Ghz, so under low thread utilization, you can expect 3+ ghz performance.

And they have 16 threads a piece. 32 threads, with the potential to run up to 3.4ghz. Not bad.

v2 CPUs (Ivy Bridge) are probably the economic sweet spot. You still get the subtantial power savings relative to Nehalem class LGA1366 CPUs (they were a space heater), you miss out on the Haswell/Broadwell generation, but those v3/v4 CPUs are the industry work-horses right now, and are just beginning to be replaced with Intel Xeon Scalable CPUs... which means they are not cheap on the second-hand market.

And Intel Scalable is a cluster-f&*k.

I would suggest you definately want the E5 class CPUs, instead of the E3. You get the cheaper ex-enterprise hardware, and you get access to gobs of ram and gobs of PCIe lanes, and the future is NVMe via PCIe.

 

[Chris Moore]

Here is to hoping I did not just step in a big pile!

I think it will work for what you want. There may be a few more components to put in here and there, like the CPU coolers, but this is a really good start.

PS. Good job finding a better deal. If I had found it first, I might have bought it myself.

 

[Chris Moore]

Just remember, it's not about air... it's about air *flow*. The heated air has to get back to the A/C return. If you seal the room up tight, you'll simply increase the static pressure of the room to the point that no air will move, and it'll be just about as bad as having no vent at all.

And yes, I've been doing a lot of thinking and studying on my closet, dealing with the same issues...q

I have been able to keep the drives at a happy temperature with an intake air temp around 77 °F. I had to go through some changes with the CPU to keep it cool enough for me to be happy but the Noctua cooler is doing the trick. The max temp in the last 24 hours was 55 °C

 

[Maelos]

I know Noctua is the go to for active CPU cooling in these chassis, but have others been tested? I don't want to cut corners and risk a CPU, but curious. I think I may have an old Artic Freezer laying around and if I don't sell my water cooling loop by the time all the new stuff gets here...

 

[Chris Moore]

I wouldn't try a water loop in a server. The reason for simple air is the reliability.
The Noctua cooling in my system runs the fan around 450 RPM and, with the airflow from the drives, it stay in the 45c to 55c range.
It doesn't need to be Noctua, but you must be cautious about the clearance and I think the socket on this board is the narrow one.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Maelos]

I was only half joking about the water cooling. I will trust in your research and results. I found them on Amazon for the cheapest, https://www.amazon.com/dp/B00E1JGFA0/ref=cm_sw_r_cp_apa_EJwLAbSZMZBKW. Of note they were $150 on NewEgg and $70+ on eBay.

I just bought two with free shipping from Amazon for $120.

Total of purchases is now $1,140. No additional drives, sound proofing, or heat management for room purchased yet.

I am looking at buying 5x more of the 8TB WD Red/White drives because it is the best $/TB. I have no idea what I would use all of that space for though. Is there any downside to extra space? Will the larger size of these drives help them last longer? Will it spread the "writes"? I'm shooting in the dark here, but I'm going to have to research this a bit more. Same goes for the need of a storage vDev alone, with a backup vDev, etc, and how all that works with the size and type of drives.

//Another link for an eventual newbie guide - https://forums.freenas.org/index.php?threads/the-math-on-hard-drive-failure.21110/

Based on the above guide, a vDev of 7x 1TB WD Red Drives ($62/per on NewEgg/Amazon, $434+) in RAIDZ3 is the safest, with a secondary vDev for backups with... 2x 8TB ($320+) in mirror? Is there even a need for a backup with such a low AFR? Should I just return the shucked drive and hunt for 1 TB drives on eBay like https://www.ebay.com/itm/LOT-5-WD-R...369296?hash=item3b032d4690:g:rVcAAOSwyRBakGxl (Currently at $40 for 5 drives or ~$8/TB). Hmmm...much to learn.

 

[Maelos]

This feels like a shameless bump, but I am here for guidance. As far as the drives go, what all do you recommend. I have about 4 TB of data psuedo mirrored on 2x 4TB WD Greens. To conserve a bit of money, I am thinking of going with 2 more 4 TB drives for the main array in Z2 and a second 8TB for a backup array in mirror. I would eventually like to make the Z2 to 6 drives, but the cost...

On the flip side I could go for 6x 8TB drives to get the biggest $/TB gain. I would then use the 2x 4TB (or more) drives in mirror to back up the most important files (files with no other backup, personal files). I have read articles on hard drive failure (link in previous post), used raid capacity planners (http://wintelguy.com/zfs-calc.pl), and read a little more do clear up why RAID and backup may be needed (https://www.intego.com/mac-security-blog/understanding-raid-for-data-storage-and-backups/).

Chris et all, what do you recommend as the minimum setup? Again, I have about 4TB of data where integrity and security > performance. I hope to open this up to friends and family as I learn to operate and secure FreeNAS better.

 

[Nick2253]

I would always recommend backups. Even the best system is not perfect. So, there's two questions here: what do you want to do for your main drives, and what do you want to do for your backups.

The reliability needs for your main array ultimately come down to your down-time tolerance. If your data is adequately backed up (again, there's no such thing as 100%, but for sake of argument here), there would be no reason to have any redundancy on your main array if you didn't care about downtime. Every time your pool when poof(!), you'd just get a replacement drive and re-download your backup, and eventually you'd be back in business.

Realistically, most people care about downtime. Furthermore, backups aren't 100%, so it's good to improve the reliability of your primary array. In that vein, the standard recommendation around these parts is RAIDZ2, because it affords you a simultaneous data error and drive failure, which is probably the most common combination leading to failure for RAIDZ1 setups.

If you are a business where any downtime is really bad(TM), or your restore times is measured in some long time period that well exceeds your allowable downtime, adding an additional redundancy drive (RAIDZ3) would make a lot of sense. For a business, the marginal cost of a couple hundred dollars is a no brainer. For a home user, it might not be worth it. Ultimately you have to decide what is appropriate for you.

I'm not exactly sure from your posts what hard drives you have, or what you're looking at buying. I would strongly recommend not buying white label drives (unless you mean that as drives you've shucked yourself). "True" white label drives are often times refurbished at best, and completely broken at worst (and that's out of the box).

Assuming you have no drives, and I was recommending a purchase from the start, knowing you have 4TB of data, and assuming you'll eventually be using another 4TB by the time you want to upgrade your system, you're looking at around 10TB (marketed) space. To get there, I'd recommend 5x or 6x 4TB drives in RAIDZ2. The 4TB drives are slightly more expensive per GB, but you only need 2x redundant drives, so you ultimately buy fewer GBs, and save some money.

For me, and my tolerance for risk, I would probably not go with RAIDZ3, because I'd rather spend that money on a higher-quality cloud backup service. Which brings us to backup options.

An option for relatively inexpensive local backup is to create a second pool (using a stand-alone or mirrored vdev), and use zfs send/receive to transfer data from the main pool to this backup pool. While this option would protect you against any kind of data loss in the primary pool, it's not really that much better than just having snapshots. It will protect you from failure of your primary pool, but it won't protect you from any other hardware failure.

A second server is a great on-site backup option, but that gets expensive. One way to work around that is get a cheap, two-disk NAS, and use rsync (or some other similar tool) to backup your main FreeNAS server. Or if you have a second server of some capacity (like a hypervisor), putting a pair of disks in there, and using that to manage the backups.

The best option is a remotely located backup. Unless you have cheap access to remote rack space, the cheapest way to do this is a cloud data store like Amazon Glacier or Backblaze B2, or a cloud backup service, like CrashPlan. The amount of data you have to back up will steer you to one or the other.