Building a "Reasonably Quiet" SSD Rack Mount Server for Office and Home use

[bollar]

Hi All,

Looking for a fresh set of eyes on my build thought process. Your ideas are welcome.

I'm going to be moving house & office in a few months and will be dealing with a couple of temporary housing arrangements over the next year meaning that I won't have the luxury of having my servers nearby, but out of the way in a server closet. The current servers are as loud as you would expect (see signature for details on those) and they are also old. It seems like there's an opportunity to build a new server that lives in the office for the next year or so without driving me crazy -- and once we're settled again towards the end of next year, move it and the older servers into the new place's server closet.

Although work and home servers are not currently combined, I think we can make do with one server for awhile. during the day, we do a lot of video editing and move large 4K files on a regular basis across a 10Gbe network. At night, mostly serving DVR shows from our collection using Plex.

Chassis: I'm aware that larger cases tend to be easier to make quiet, but since we will be moving several times, I've decided on a 2U chassis, using SSDs. I already own the chassis -- a SuperMicro SuperChassis 216BE1C-R920LPB. It's 2U, SAS3 and holds 24 SSDs in the front and 2 additional SATA SSDs in the back. It supposedly has a "Super Quiet" redundant power supply. I may put some case insulation in the chassis, though I hear that maybe it doesn't do all that much.

CPU: I think Intel Quick Sync video will eventually be useful, but seems like it doesn't make sense to turn away from Xeon. So, that's a very short list and I'm looking at the Intel Xeon E-2388G Processor (thanks to @jgreco for pointing out this chip).

Motherboard: With the E-2388G as the CPU, that leaves a very small list of SuperMicro motherboards -- the X12ST series. I think the X12STH-F is the best choice there. It's Micro-ATX, so should fit fine in the cavernous chassis. It has six SATA ports. Two are used for the rear drive slots, leaving four for something. It also has an M.2 slot.

CPU Heat Sink: Not sure what to do here. SuperMicro recommends a passive heat sink, but I usually put a Noctua in my builds -- however, Noctua says their CPU fan is not compatible with this motherboard due to something on the board that's in the way. SuperMicro also sells a 2U heat sink with a fan -- looks like it would be trivial to replace the fan with a Noctua if I wanted. Also thought about liquid cooling, but I don't know anything about it. Seems like the 2U might be too tight anyway.

RAM: 128GB (the max this board takes). Basically 4 x Crucial 32GB DDR4 3200 MT/s CL22 DIMM 288-Pin, as recommended.

HBA & Networking: Unless someone suggests otherwise, I planned to stay with the SuperMicro-recomnded cards: Supermicro 12Gb/s Eight-Port SAS Internal Host Bus Adapter (which is an LSI SAS 3008) and AOC-STGF-i2S (Standard Low-Profile Dual-port 10GbE controller with 2x SFP+ ports).

SSD: I haven't decided what to do here. To some extent, it depends on how much I decide I "have-to-have" immediately available (instead of firing up one of the old servers temporarily, assuming it's accessible). Probably I'll go with whatever brand name is cheapest at the time -- like the Samsung 870 QVO Series 2.5" SATA III 4-bit QLC V-NAND. I also don't know the best practices on pool and VDEV design for SSDs, so will go research that. I'm sure there are existing threads on that topic.

So, that's my current thinking. Have I missed anything? Should I look at something else? I have a couple of months to get this sorted and even though I have the chassis, I could replace that if I absolutely had to.

 

[ChrisRJ]

The 870 QVO are not suitable for write-intensive workloads. So for serving files to Plex they might be ok, but not for video editing.

 

[Davvo]

The thing that's in the way is likely the backplate glued on the motherboard.

[Official] CPU Heatsink Recommendations

Air Cooling Notes Generally speaking, Arctic makes a product that works for every solution. They are affordable, high performance, quiet, and well built. LGA115X = LGA1156, LGA1155, LGA1150, LGA1151, LGA1200 LGA115X (hole/backplate) Consumer motherboards usually have pass through holes. Xeons...

forums.serverbuilds.net

 

[bollar]

The 870 QVO are not suitable for write-intensive workloads. So for serving files to Plex they might be ok, but not for video editing.

Thank you. That's an important consideration I need to work through -- perhaps I should create an SLC pool for the video editing tasks. But wow, that's eye-blistering prices. :D

 

[bollar]

The thing that's in the way is likely the backplate glued on the motherboard.

[Official] CPU Heatsink Recommendations

Air Cooling Notes Generally speaking, Arctic makes a product that works for every solution. They are affordable, high performance, quiet, and well built. LGA115X = LGA1156, LGA1155, LGA1150, LGA1151, LGA1200 LGA115X (hole/backplate) Consumer motherboards usually have pass through holes. Xeons...

forums.serverbuilds.net

I think you're right. That's a great link -- thank you!

 

[jgreco]

thanks to @jgreco for pointing out this chip

Yer welcome. Nice to know that the immense amounts of time I spend researching stuff benefits other folks too. Also would be happy to hear how it works out for you.

 

[bollar]

I've never really had to think about performance on the old system -- slow spinning drives were always my limiting factor.

But now, I'm thinking about it and have done some reading. Unfortunately many of the most popular threads are ten-plus years old and may or may not be pertinent. Here some conclusions and more questions:

- There's no need for a SLOG or L2ARC. I get some benefit from a SLOG on the old system, but it's not huge. I assume it would approach zero when mated with SSDs.
- I'm uncertain if I would benefit from a metadata VDEV. It would have to be on SATA, so less potential throughput. I could do striped & mirrored SLC drives, which seems like it would take some stress off the data VDEVs, especially if I go the cheap route. But, that would take six drives to match the parity on the data VDEVs, which seems like a lot.
- Similarly, I could have a separate mirrored pool of SLC drives for the high write activity tasks. That would be on SATA as well, so there is a potential performance consideration.
- Boot will be a mirrored pair of SATA DOMs.
- Is there anything I could do with the M.2 slot? I thought about an M.2 Optane to use for transcoding / scratch.
- Older threads I came across (pre 2015-ish) had horror stories of entire batches of SSDs failing at pretty much the exact same time. I assume we're well past that now. Still, probably no harm in getting comparable drives from different manufacturers.

Assuming I create a single pool from the twenty-four slots on the SAS3 backplane, what are the optimal configurations? I don't think I can bring myself to go mirrored, despite the obvious potential performance benefits. If the considerations are basically the same as for spinning drives, I see the choices as 8 drive Z2 VDEVs x 3 or 12 drive Z3 VDEVs x 2. Maybe statistically, the Z3 is more resilient, but Z2 seems like it has more benefits. I wouldn't be comfortable using Z1, even though some said that SSDs don't need the same resiliency that spinning drives need.

Some of this might be leading edge for SMB servers now, but I was able to keep the old server going for ten years with only small upgrades, and it was pretty advanced then. If I can get ten years from this server, I'll be thrilled.

 

[Davvo]

- There's no need for a SLOG or L2ARC. I get some benefit from a SLOG on the old system, but it's not huge. I assume it would approach zero when mated with SSDs.

SLOG: you always benefit from it when you have syncwrites, you don't benefit from it when you have asyncwrites. The performance improvement you get from HDDs is greater compared to SSDs, but you always have an increase of performance (because you have a dedicated drive).

L2ARC: you benefit from it when you have at least 64GB of RAM, and it always helps.

METADATA VDEV: generally useless (more like the bad alternative) until you haven't maxed your RAM and L2ARC or have very specific uses (ie deduplication).

Pool layouts for maximum efficency are something worth spending time studying: what's your objective? Having a separate pool made of high endurance SSDs would be helpful if you have the needs of working on the footage.

 

[ChrisRJ]

My gut feeling tells me that SLC SSDs are not necessary for your use-case. MLC or good TLC would be my instinctive choice for video editing. I would think hard whether or not TLC is still appropriate for a database server that is under heavy load 24x7. But that is entirely different workload from what you described.

 

[awasb]

I‘d say in this use case it mainly depends on the (business) value of the data (and less on the workflow). If it‘s business money you‘ll burn with a drive failure (and subsequent data loss), I‘d stay away from DRAM caches, SLC driven „performance areas“ and dubious controller background optimization. If it gets interrupted …

You can‘t compensate that risk with any SLOG or UPS. If atomic writes are forcefully aborted, your metadata (and in the worst case your pool) will be toast. ZFS won‘t „know“, since all that crap happens in the drive(s).

If it is business critical, I‘d buy enterprise/datacenter drives (with PLP) any time.

 

[bollar]

My gut feeling tells me that SLC SSDs are not necessary for your use-case. MLC or good TLC would be my instinctive choice for video editing. I would think hard whether or not TLC is still appropriate for a database server that is under heavy load 24x7. But that is entirely different workload from what you described.

Thanks for those tips. I don’t want to overstate what’s going on with the videos. Currently, we copy the 4K / 8K work files from the server each morning, do all the processing and then copy them back at the end of the project or end of the day. So, it’s really just a few TiB a day being written to the server.

But, if Quick Sync works, I could see maybe transcoding on the server. It’s the kind of process that I’d be happy to let run overnight.

I’ve used a consumer SSD that I repurposed as a scratch drive and that lasted only a couple of months before I blew through its lifecycle. If I decide to try it out, I’d like to avoid that happing again.

 

[bollar]

Pool layouts for maximum efficency are something worth spending time studying: what's your objective? Having a separate pool made of high endurance SSDs would be helpful if you have the needs of working on the footage.

Thanks!

I don't think of us as intensive server users right now. It's not much of an oversimplification to say that we download the large files once a day in the morning and upload edited files once a day in the evening. Since they're large, faster is better, but we've been "making do" with up-to 3Gbps-ish transfer speeds for years now.

Everything on the business side is fully backed up or can be replicated. RAW files are on SSD and retained well through the end of an engagement. Complete server backups are collected from us weekly and stored off-site. Each workstation also has its own backups to SSD and we keep about 1 TiB of data at rsync.net. So, I'm not particularly concerned about losing anything forever But, as @awasb noted, the time it would take to rebuild a server would be long and it doesn't pay the bills. On the home side, don't really care -- it's all Blu-ray and OTA recordings off our antenna. If the show is any good, it will be aired again.

So, objectives:

- Needs to provide comparable data security to what we currently have (which is 3 VDEVs of Z2).
- Must be "relatively quiet."
- Would be nice if file transfers were faster.
- Would be nice if transcoding with Handbrake (or whatever) is possible.

 

[bollar]

Ordered most of the parts aside from the SSDs and we'll see them show up in the next week. There are a number of open box items on eBay and NewEgg, so I was pretty pleased with the pricing. Well, except for the Intel E-2388G, which was way more expensive than I expected -- almost $700.

Also got an Intel Octane 905P for the M.2 slot. I guess I could use that for SLOG or scratch, but I haven't done the research on that yet. Seems like less than 10% of our traffic is sync writes, so I'm not sure how much benefit we'd feel immediately.

 

[Davvo]

You don't need a SLOG imho.

 

[bollar]

Based on your gentle suggestions on SSDs, I decided on mirrored Samsung Pro 850 VDEVs for the work and ix-applications pools. I'll stick with the Samsung QVOs in Z2 for what I think of as "warm storage."

 

[bollar]

For those who might find this thread by search, the Supermicro X12STH-F motherboard has a backplate, but it's not usable for attaching the CPU heat sink. Rather, it has the open holes like one would expect on a consumer motherboard. The Supermicro-recommended passive heat sink (SNK-P0049P) needs a backplate and I did not find a Supermicro part number for that. So, I ordered a generic consumer backplate for the LGA-1200 (and LGA-115x) and we'll see if that works.

If it doesn't, there are some active heat sinks that look like they would work. Most likely is the Arctic Freezer 11 LP. It has a three-pin fan that uses its own sensor to determine fan speed -- it doesn't use the IPMI settings. Arctic calls it silent, so maybe it doesn't matter.

Otherwise, parts have been arriving all week. All the eBay & Amazon stuff came quickly -- the things I ordered from Newegg haven't been so fast -- still waiting on some that might not arrive before the end of the week. Not that I'm in a hurry...

 

[Davvo]

Never seen that kind of backplate. It looks like you can install commercial coolers there.

 

[joeschmuck]

Never seen that kind of backplate. It looks like you can install commercial coolers there.

That backplate looks like the CPU Socket backplate, not a CPU fan backplate. There are definitely holes many cooler types.

 

[bollar]

That backplate looks like the CPU Socket backplate, not a CPU fan backplate. There are definitely holes many cooler types.

Yeah. I guess this combination of Xeon and pushpin holes adds some complexity. Several vendors, including Noctua, use reinforcement on the bottom of the board and the CPU backplate interferes with their solution. Anyway, the Arctic Freezer 11 LP arrived today and installed correctly.

Now just waiting on the RAM, which is apparently on Pony Express from Illinois. ‍ I hope it works -- Supermicro says you can populate the board with 128GB, but only have approved RAM on their list to 64GB. Lolz.

 

[jgreco]

Supermicro says you can populate the board with 128GB, but only have approved RAM on their list to 64GB. Lolz.

This is very common. The lifecycle of many boards is five to seven years, and Intel usually designs the CPU's to go at least one power-of-two larger than the largest commercially available memory module at the time the CPU is released. This means that two to three years later you will see those 128GB DIMM's at "holy crap that must be plated in gold" prices, followed by the five to eight year pricing where it's a tenth of what two of the 64GB'ers originally cost.

The tested compatible memory is mostly advisory in nature. Supermicro designs their systems to work with generic memory, but the paranoid among us may seek the additional reassurance of knowing that someone put it to the test under lab conditions and hopefully checked out the memory bus signal quality of the "recommended" modules with a 'scope. This is different than other manufacturers who lock you into very specific modules.

So it makes sense, but only if you hang around long enough to be interested in maxxing out the memory at a later date.