24 Bay Build Questions

[Hinatanko]

I am looking for advice on a FreeNAS build that I would like to have built in the next couple months. I was able to build a smaller 8 bay build last year without any problems or questions based off of all the information on these forums. However, while their is a lot of information, I want to make sure I make no mistakes since this build will cost atleast $5-6k starting out. Below is information on how I will use the server, the parts I have pick so far, and any questions I have.

Intended Use
This server will mainly be a storage box for videos, music and images. I will be using Plex Media Server and do not plan to have more than 2 1080p streams going concurrently. I may also run a few other jails in the future for downloading files.

Hardware

Hard Drives
12x 6TB or 8TB WD Red or Seagate NAS Drives

I am waiting to see if either of these goes on sale. I would prefer to go with the WD Red as long as the price is right. My plan is setup one vdev of 12 HDDs in RaidZ2 and buy another group of 12 in the future. I have seen people on here make vdevs with 6 HDDs in RaidZ2 and some have done 8 HDDs in RaidZ2. I would prefer to do 12 in RaidZ2 to get more space, but obviously since I am using FreeNAS I do not want to worry about data loss. Is doing 12 HDDs at 6TB or 8TB in RaidZ2 safe or should I do 8 HDDs in RaidZ2? I also know their is RaidZ3 is this something I should look into? I know this is probably more of a personal choice, but your feedback is greatly appreciated on this.


Chassis
Supermicro SuperChassis 846E16-R1200B
http://www.ebay.com/itm/SUPERMICRO-...S-WITH-24x-TRAYS-INCLUDES-RAILS-/381077493540 (old auction but something like this)

Currently I have not seen any auctions that seem like the kind of deal I am looking for. I was hoping to pay less than $500 for this case on ebay. The best deal I have seen is $600 from MrRackables.

From my understanding when researching this, it seems like the SAS2 backplane is a must for modern hardware (not sure if SAS3 is needed or not, correct me if I am wrong). It also seems the E16 version of the backplane is the easiest way to go. I am not very familar with SAS expanders or HBAs in general. My only knowledge is from the pinned topic on here. This seems like this would be the easiest solution.


Motherboard
Supermicro X10SRL-F or X10SRi-F or X10SRH-CLN4F
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182927
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182928
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182931

My choices for motherboard are based on the recommendations on here. I was considering either X10SRL-F or X10SRi-F but saw that X10SRH-CLN4F has an LSI SAS controller that might allow me to skip buying any addon cards for the backplane (I may be wrong about this). The prices are very similar between X10SRL-F and X10SRi-F so I am not sure what you would recommend between those two. I read the differences in the recommendation thread and they seem very minimal for my use case. What motherboard would you recommend overall?


CPU
Intel Xeon E5-1650 v3 or E5-1620 v3
http://www.newegg.com/Product/Product.aspx?Item=9SIA1UH3Z95626

The prices are about half (based on Newegg), but I am not sure how big of a performance difference I will notice. I looked at benchmarks for both as well. Will FreeNAS need the extra CPU power because of how large the build will be? Are these the two CPUs I should be looking at or is another CPU I should consider?


RAM
8x Hynix HMA42GR7MFR4N-TF DDR4-2133 16GB
https://www.amazon.com/dp/B00Q6QOLJ8/?tag=ozlp-20

If I end up getting 8TB drives eventually I will have 192TB of raw capacity. I have always assumed the 1GB per 1TB of space is based on the raw capacity.

I would like to buy enough RAM now so I do not have buy anymore in the future. I read a thread where adding bad memory caused problems to a server. Is this something I should have to worry about?

Is 128GB of RAM not enough for 192TB or 144TB (if I get 6TB)?

The RAM I listed is from Supermicro's list of memory that will work. Does this look to be the correct ram? If I need to go with more than 128GB I have looked at this: https://www.amazon.com/dp/B01HMSSVUS/?tag=ozlp-20

Is this RAM also look and safe to use? It is also from Supermicro.


HBA
IBM ServeRaid M1015
https://www.amazon.com/dp/B0034DMSO6/?tag=ozlp-20

I am a novice when it comes to this. I have read quite a bit about this, but since I have never dealt with any HBAs before I want to make sure I have this part correct.

If I get the X10SRL-F or X10SRi-F (and not the X10SRH-CLN4F) will I need only one of these as long as I get the E16 version of the chassis I listed above? Also what kind of cables and how many would I need for 24 drives? I believe the cable is SPF 8087 and I will need atleast 2-4? Any help with this is greatly appreciated.


Fans/Noise
This server will be in my bedroom temporarily for about a year. I understand I will have to deal with noise. Any suggestions on fans or other ideas to reduce noise is appreciate.

From what I read the fans in Supermicro Chassis should work just fine. Do I need any additional fans besides a heatsink for the CPU?


Boot Drives
In my previous build I used a SATA DOM. I am reading that I should probably go with an SSD this time and will need two of them to mirror. I read that any cheap SSD from a good brand should work. Do you have any suggested brands and sizes for my build?


Rack Cabinet
Norco C-24U 24U Rack Cabinet
http://www.newegg.com/Product/Product.aspx?Item=N82E16816133052

I am not really confident in this choice. I read that Craigslist is a good choice, but I live far away from a large city. I was looking at eBay but did not see anything in particular. Do you have any recommendations for Rack Cabinets? I know Norco is not recommended for Chassis. So I am not sure if thats the case for cabinets as well. I am wanting to go with 22 to 24u for future expansions. I was planning on sitting my Synology DS2415+ at the bottom of the cabinet since I can't mount it. I don't think this will cause a problem. If it may let me know.

UPS
Any good rack mounted UPS recommendations?

SLOG & L2ARC
I read about this. Since I am going to have more than 64GB of RAM will I need an L2ARC for the use case of this server? Same with SLOG. If so what would you recommend?

Prebuilt System From MrRackables
http://www.ebay.com/itm/Supermicro-...e-192GB-2PS-/142173205659?hash=item211a2ef49b

I have read that people have had good luck with MrRackables. When looking for a deal on the Supermicro Chassis I saw the chassis I wanted but with more expensive hardware. Do people ever buy the more expensive servers from MrRackables and keep the hardware? I have only seen threads where people recommend the chassis and get rid of the hardware. The reason I ask is because the listing above appears to have everything I need outside of the drives. Would you ever recommend going this route when spending this much money. I understand the hardware is used. I rarely buy used products so I am not experienced enough to know if hardware components from a reliable source is a good idea or not.


Sorry for the long post. I am sure I have made several mistakes so any help would be greatly appreciated.
Thanks!

 

[Ericloewe]

Will FreeNAS need the extra CPU power because of how large the build will be?

Probably not. @jgreco has some pretty intensive storage needs running on a Xeon E5-1650 v3 and the thing doesn't break a sweat.

I have always assumed the 1GB per 1TB of space is based on the raw capacity.

It's not. It's intentionally vague.

Is this something I should have to worry about?

No, just get high-density DIMMs (for ease of upgrading later) and stick to the QVL.

The RAM I listed is from Supermicro's list of memory that will work. Does this look to be the correct ram?

Yeah, but have a look at 32GB RDIMMs instead. They'll leave half the DIMM slots empty for now, making upgrades easier. Price shouldn't be very different.

will I need only one of these as long as I get the E16 version of the chassis I listed above?

Yeah.

Also what kind of cables and how many would I need for 24 drives? I believe the cable is SPF 8087 and I will need atleast 2-4? Any help with this is greatly appreciated.

One SFF-8087 cable if you're using an SAS2 HBA like the M1015, one SFF-8643 to SFF-8087 cable if you're using an SAS3 controller like the one on the X10SRH-C/-CLN4F.

This server will be in my bedroom temporarily for about a year. I understand I will have to deal with noise. Any suggestions on fans or other ideas to reduce noise is appreciate.

Earplugs. And make sure you get a modern PSU with temperature-controlled fan, otherwise those earplugs won't do you much good.

rom what I read the fans in Supermicro Chassis should work just fine. Do I need any additional fans besides a heatsink for the CPU?

You need a CPU cooler, probably best to get a desktop-like one for noise reasons. Noctua has a few server-oriented coolers which are good.

need two of them to mirror

Not really, one is okay. Overkill is fine, though.

Do you have any suggested brands and sizes for my build?

SanDisk, Toshiba, Samsung, Crucial, Micron, ...

 

[Ericloewe]

Bah, posted early and editing that is going to be a pain...

Any good rack mounted UPS recommendations?

An APC Smart-UPS within your budget and power needs.

Since I am going to have more than 64GB of RAM will I need an L2ARC for the use case of this server?

There's some confusion there. In any case, try it out first and then add L2ARC if your ARC stats suggest it would help.

Same with SLOG.

No.

 

[Stux]

And consider v4 versions of your CPU choice if the price is the same.

Re 1650 vs the 1620. Yes, twice the price, 50% more cpu power. That's how intels pricing works unfortunately :(

If you realize that for an extra xxx$ you can increased the CPU performance of your entire xxxx$ system then it might make more sense

I assume you've seen my build?

 

[Hinatanko]

Thank you for the feedback!

Below is a list of the hardware I plan to use based off your recommendations so far:

Chassis: Supermicro SuperChassis 846E16-R1200B
Motherboard: Supermicro X10SRi-F
CPU: Intel Xeon E5-1650 v4
Cooler: Noctua NH-U9DX i4 (2 x Noctua 90mm NF-B9 PWM fans)
PSU: Included PSU from Supermicro Chassis
RAM: 4x Samsung 32GB DDR4-2133 Memory MEM-DR432L-SL01-ER21 M393A4K40BB0-CPB
Boot: 2x SanDisk SSD PLUS 2.5" 120GB SATA III MLC
Storage: 12x 8TB WD Red or Seagate NAS HDDs (2 vdevs of 12 HDDs in RAIDZ2)
HBA: IBM ServeRAID M1015
Fans: Included Fans from Supermicro Chassis
UPS: APC SMC1000-2U Smart-UPS 1000VA 600-Watt 120-Volt 2U RM LCD USB with Uninterrupted Power Supply
Cabinet: Norco C-24U 24U Rack Cabinet

All of seems fine from what I can tell. Let me know what your thoughts are on this.

From everything I read it seems like its better to go with the Supermicro Chassis over the Norco. The only advantage I have heard of is its not as noisy.

Is a 12 HDD vdev with 8TB drives in RAIDZ2 safe or should I look at something different?

Should try to find a used cabinet on ebay instead of buying the Norco one?

Thanks again!

 

[Dice]

Fans/Noise
This server will be in my bedroom temporarily for about a year. I understand I will have to deal with noise. Any suggestions on fans or other ideas to reduce noise is appreciate.

Do you have any experience with server grade components in a normal home environment (aka - where ambient noise is low?)

 

[Hinatanko]

Do you have any experience with server grade components in a normal home environment (aka - where ambient noise is low?)

Nothing like this. I have a Synology DS2415+ NAS and a FreeNAS box with about 4 fans in it. The FreeNAS box is in a tower chassis and all of the parts recommended on here for 8 bay builds.

Noise isn't my main concern, but I know people have mentioned that it can be surprised how long it can get. So I wasn't sure if their something I should know to help.

I know the Supermicro fans are suppose to be loud but good quality and the chassis has better airflow than the Norco. I wasn't sure if people replace the fans in Supermicro chassis with quiet fans since theirs better airflow. From what I have read it seems like the PSU makes a ton of noise.

 

[MatthewSteinhoff]

Prebuilt System From MrRackables
http://www.ebay.com/itm/Supermicro-...e-192GB-2PS-/142173205659?hash=item211a2ef49b

We have purchased equipment from MrRackables and were happy with the results. We always try to buy used or refurbished - more than $8,000 in savings his year - instead of new and have yet to be burned. Cost savings are substantial.

If that listing is what you need and less expensive than purchasing new, I wouldn't have any hesitation in purchasing the complete unit.

Cheers,
Matt

 

[Hinatanko]

We have purchased equipment from MrRackables and were happy with the results. We always try to buy used or refurbished - more than $8,000 in savings his year - instead of new and have yet to be burned. Cost savings are substantial.

If that listing is what you need and less expensive than purchasing new, I wouldn't have any hesitation in purchasing the complete unit.

Cheers,
Matt

Thanks for your feedback Matt. The CPU and Motherboard are similar enough on that listing. I would only need to add boot drives and the HDDs of course from what I can tell. Without UPS or Cabinet and HDDs the listing would be ~$1700 with an SSD or two added. To buy brand new with the parts I listed above it will be $1900+the cost of the chassis.

The chassis has been the hardest part to find at a great deal. I was hoping to spend $300-$400. Right now I am only seeing the E16 SAS2 version of the chassis I want for $900 and up at the moment. If I can find a good deal I may pay the extra few hundred for new hardware, but if not I may consider going with the MrRackables server unless someone has had a bad experience. If anyone knows of a good deal on the Supermicro Super Chassis 846E16 under $500 I would appreciate it. I haven't had any luck so far. I've only started looking in the last couple of days though.

I would like this server to last 10 years if thats realistically possible. In the future I want to be build a better backup system that is similar to this server.

 

[Dice]

Nothing like this.

There is no way you'd cope with the noise over a period of time.
My threshold before I start reaching for earplugs (just to do maintenance on the machine) is about half an hour.
After 2-3 days of ear plugs in my home I went pretty much nuts.
No matter how steep the investment was, the machine is still sitting shut off, while my previous box is doing the duties.

Coming from a synology box and a freenasbox with "4 fans" will not prepare you for what you're looking at.
SMs are GREAT for window shopping and general tech giggles... but IRL ...the story is different.

https://forums.freenas.org/index.ph...buying-an-old-server.44666/page-2#post-300606

 

[Ericloewe]

Hey @Dice - does your system have the temperature-controlled PSU fans? I was wondering if the noise from the chassis fans (under typical conditions) would be bearable, kept away in a different room.

 

[Hinatanko]

Is their any alternatives since Noise level seems to be an issue? I am not completely against Norco, but I really liked the idea that it came with a PSU.

Any suggestions on this will help me determine the chassis.

At this point I feel pretty confident on the rest of the hardware and just unsure on how to do my pool.

 

[Dice]

Hey @Dice - does your system have the temperature-controlled PSU fans? I was wondering if the noise from the chassis fans (under typical conditions) would be bearable, kept away in a different room.

The original PSU's have temperature controlled fans.
My single replacement PSU is significantly less noisy.

I'd definitely say the box can be tweaked to become bearable in another room - granted there is a shut door and decent walls.
There is no such thing as putting it into a closet in the master bed room and calling it done. That is just not even remotely close to an option as far as my experience goes.
In contrast I've a 7 drive X11 box that is, after some tweaks, basically inaudible in a closet. I am cranky when it comes to noise. But not impossible.

As soon as I revisit the box for another attempt at making it bearable in my 1 room appartment, I'll probably put up a thread sharing the experience in a bit more detail as opposed to my scattered comments in various threads.

 

[skyline65]

Watch this... and listen to the noise, could you sleep in a room with this?
https://www.youtube.com/watch?v=Ym0uHos09Cc

Have a look at Stux’s build as he swapped fans over and created a script for controlling the fans:
https://forums.freenas.org/index.ph...24-supermicro-x10-sri-f-xeon-e5-1650v4.46262/

 

[depasseg]

Watch this... and listen to the noise, could you sleep in a room with this?
https://www.youtube.com/watch?v=Ym0uHos09Cc

Just an FYI - that's not the PSU fan's it's the system fans running full speed. I moved my fan connections from the backplane (which doesn't have temp/speed control) to the motherboard, which does, and it makes a WORLD of difference.

 

[MatthewSteinhoff]

Supermicro servers are loud. It doesn't bother me because mine live in the server room. It's always loud in there. I wouldn't recommend them - or HP servers, or Dell servers - in a residential environment. Especially if you also sprung for the Spouse upgrade.

Yes, you can jump through hoops to reduce fan speeds but then you're faced with having to supercool the room in which the server lives to compensate for the lower fan speeds. Not worth the hassel.

Cheers,
Matt

 

[Stux]

My Norco build is quieter than a drobo.

 

[Hinatanko]

My Norco build is quieter than a drobo.

I'm debating whether I can move around some things and put the server in a different room that no one is in or not. If that doesn't work I am looking to basically copy your build.

Does anyone think 12 drives in a vdev in RAIDZ2 or RAIDZ3 is a bad idea or would you recommend 3 vdevs with 8 drives in raidz2? If both are viable options I would probably prefer to do 12 drives in raidz2.

 

[SweetAndLow]

IMO no way you could spend extended periods of time in the same room as a supermicro server on the lowest fan settings. Mine is in the crawl space under house and i can hear it all the time.

 

[anodos]

IMO no way you could spend extended periods of time in the same room as a supermicro server on the lowest fan settings. Mine is in the crawl space under house and i can hear it all the time.

On the other hand, once you have kids it's not a problem. 80DB? That's called white noise.