Experiments in small form factor + low power

[tarnar]

Hi all, wanted to share my build log of a new experimental zfs storage server. I'm replacing an older system that I totally overbuilt (more on that later) and had the following objectives:

1. Reliable storage for high priority backups
2. Small size
3. Low power draw
4. Low noise

So here's where I started

Part List

Rosewill RSV-Z2600 Short Throw 2U Case w/ 80mm Fans, (supposedly) room for 4x3.5" Drives
SuperMicro X11SSM-F
Xeon E3-1230 V6
Noctua NH-L9I CPU Cooler
Kingston KVR24E17D8/16 ECC RAM
Corsair SF450 Small Form Factor Power Supply

I chose to replace the case fans with 80mm Noctua PWM fans (quiet!)

The HBA is an eBay'd LSI 2308 (specifically a Dell 9217-8i). SATA breakout cables to 8 drives.

The drives are where this gets a little interesting. I went hunting for cheap m.2 SATA solid state drives, and found four Crucials and four Sandisks, all at the 500GB-ish size.

Those drives sit on a converter card that seats four m.2 SATA drives in a 3.5" form factor. The specific card I went for is from an outfit called MicroSataCables.com

Then there are two boot drives (2.5" SATA SSDs I bought from a friend) in a 2x2.5" to 3.5" StarTech bracket.

The Gotchas

Where to begin!

The power supply didn't come with a bracket to fit standard ATX. I specifically chose an SFF power supply to have more room, and I'm glad I did.

SuperMicro has their backwards case/motherboard/power supply design, so the power connector is against the side of the case, and I needed an extension to attach it.

The HBA ports are right in the way of the middle 3.5" drive mount, so that one has been removed.

The initial plan was to have the two 3.5" converter cards on the side of the case where you can put two 3.5" drives on top of each other. That didn't pan out, so the cards are where they fit - on each side of the case.

The power supply has connectors for old-style hard drive power (4 pin molex) but the converter card layout demands right-angle molex connectors in this form factor.

The HBA gets very toasty under load and the case is lacking in great airflow so I elected to tape a 40mm Noctua fan to it the heatsink on it. In theory I have a side-mounting bracket to hang the fan from instead of tape, but it's not here yet.

The Old Build

The old system was a 2U SuperMicro chassis with dual power supplies, 8x3.5" drives and an X9 board. It was loud and I never had anywhere to run it that I felt good about in my home.

I also had imagined that I needed many, many TBs of storage, but that wasn't my actual use-case. I don't keep a home media library, and in the end all I really wanted was somewhere to keep my backups and mirror my friend's backups. The old build was virtualized (ESXi) but that virtualization didn't do much for me much in the end.

So this time around the goals were focused on what I saw as the old weaknesses - size, form factor and noise.

The Final Build (hardware)

First, the m.2 to 3.5" card empty

Then the cards populated with SATA SSDs

Then the cards mounted in the 3.5" bays

A shot of the rig during testing (before my HBA arrived)

And finally, the finished product.

Maybe more to come. I'm still experimenting with the OS loads (have played with Linux and FreeNAS so far) and not sure if I'll virtualize this time around.

Let me know if you have questions. This has been a very fun project, and I'm planning to build another one of these. This version was a test-run that I'll also be able to use for my backups, and I hope to apply the lessons learned to another build soon.

Edit: this time with working images..

 

[John Doe]

very interesting build. could you share some test results?
power consumption?
read write
...
usable TB space

just a question, the HBA consumes ~18w the 2 ssd drives maybe 3-5w so by using 2 flash drives and connect the m2 ssds to the c236 chipset, you could safe a lot of energy. Most probably you have a reason for this. would you like to share it?

 

[tarnar]

very interesting build. could you share some test results?
power consumption?
read write
...
usable TB space

Write performance is 1.8 GB/s from /dev/zero, but I need to retest to be sure I'm doing it right.
What's a good way to measure read performance?
Still need to measure power consumption, I have a meter but haven't got around to it yet.
It's in a raidz2, for reasons that make more sense related to the below question.
Usable space, I will update to add that.

just a question, the HBA consumes ~18w the 2 ssd drives maybe 3-5w so by using 2 flash drives and connect the m2 ssds to the c236 chipset, you could safe a lot of energy. Most probably you have a reason for this. would you like to share it?

While I said above that virtualization isn't that important to me, I also didn't want to take that option away. If the system is quiet enough then it'll sit next to my "network stack" (aka the cable modem + home theater setup) and in a virtualized setup I can put pfSense on another VM and get the right VPN setup going for mutual backup sharing.

And while it's definitely overkill, I'm playing it extra safe here.

raidz2 means double parity, means I can scrub with a failed disk. In a two-way mirrored configuration I can't scrub with a failed disk, and a triple mirror feels like a bit of a waste. On top of that, the 500GB m.2 SATA drives were a particularly attractive price/performance ratio, and that ratio wasn't there for larger drives. The performance gains from an 8 drive array aren't important, especially since I don't have 10GbE in there (yet...)

My power consumption concerns are mostly noise related. This power supply won't spin up below a certain load, and is not spinning on the photos posted (note the blur on the CPU cooler and not the power supply). It doesn't spin up when I put full load on the CPU, but haven't tested full CPU + HBA load.

Speaking of noise, one of my problems right now is with PWM control of fans on this motherboard. This forum has a lot of great resources for controlling fan speed, none of which appear to work as expected. I intend to connect FANA (which should be it's own zone as far as the IPMI is concerned) into a PWM fan hub/multiplier, and have the front fans controlled through it + the CPU controlled through FAN1 (a different zone). Even without the hub/multiplier I simply cannot get it to control the speed of FANA. Still need to go back and try again.

I was able to stop the IPMI from freaking out over low fan speeds - again using the excellent resources on this forum - which was good to solve, as the fans would throttle up/down endlessly.

 

[tarnar]

The system above was put aside for a little bit while I worked on another version. This one wasn't for my home use, so it had some different objectives:

Form Factor - needs to be two-post rackmounted
Bigger Capacity - needs to backup 30-ish laptops
Low(ish) Power + Heat - resides in a network closet, not a full-fledged data center

Part List

Athena Power RM-3UC338
Supermicro X11SSL-CF-O
Xeon E3-1270 V6
Kingston KVR24E17D8/16 ECC RAM x 4
Corsair SF450 Small Form Factor Power Supply
ICY DOCK 4 x 2.5 SSD to 5.25 Drive x 2https://www.newegg.com/Product/Product.aspx?Item=N82E16817994181
Micron 1100 2TB 2.5" SSD x 8https://www.newegg.com/Product/Product.aspx?Item=9SIA8TK78R3886
4 Port Intel Gb NIC
Kingston A400 120 GB 2.5" SSD x 2 (boot drives)

The Build

The previous build helped out here, this one was a lot smoother. 3U is a lot easier to deal with than 2U.

I stuck with the SFF power supply + bracket because the extra space is nice. Still needed an ATX power extension cable.

The motherboard built-in HBA makes the build cleaner, and the SAS connector is clear of the 5.25" bays, so the (relatively) tall connector fits fine.

The price on those Micron 2TB SSDs in Canadian dollars works out to 5 GB per dollar. That just feels absurd!

With an extra SATA power splitter the bays and boot drives run off a single power cable from the power supply. The short lengths of the SFF power supply cables is definitely a drawback.

The Use Case

A bunch of GbE laptops need to do an additional backup (besides local backup to a USB harddrive at the desk). The 4 port GbE will be bonded together so that multiple machines can backup at the same time.

The 8x2TB drives go in a raidz2.

Unlike the previous build, noise isn't a concern. It goes on the network rack, which is in a climate controlled riser room. So the stock cooler stays, as do the case provided fans (they're noisy little things).

Some Pictures

The front, showing the two bays. Boot drives not yet installed.

Close up of the SATA + SAS connectors, showing the nice clearance between them and the 5.25" bays

Top view before installing boot drives or attaching SAS->SATA cables. Boot drives go into the right-hand 2.5" bay, so their connectors don't interfere with the motherboard connectors. Both 2.5" drives fit into the single bay!

A little closer to the final build. Still needed a SATA power Y splitter for the boot drives, so they're just hanging out right now.


Overall, this was a much more pleasant machine to work with. Hot swap drives is a nice perk too.

 

[Inxsible]

Form Factor - needs to be two-post rackmounted
Bigger Capacity - needs to backup 30-ish laptops

That's a problem.
If you need big capacity.. you'd need more drives which increases the weight of the machine/chassis. And putting that on a 2-post is not good. I have a 2-post rack which houses my switch, patch panel, PDU and my pfSense router, but I am following this DIY guide to build my own rack enclosure with a 4-post in order to put in my new 12 drive chassis.

Once built, I will be moving my current equipment into it as well and sell the 2-post rack. My current NAS - which is not rack-mount -- will become my backup and stay in a separate room.