First Build, 836BA-R920B and X9SRH-7F-O, 26TB raidz3

[DearestDreamer]

Greetings fellow enthusiasts,

I'm creating my first storage/media server and I'd really appreciate a second opinion on this build from someone who has more experience, especially with Supermicro components. Here it is:

* Case:
SuperChassis 846A-R1200B
SuperChassis 846E16-R1200B
SuperChassis 846A-R920B
////////// The 24x SuperChassis are too expensive ($1000+)
NORCO RPC-4220 4U | $330
~
120mm fan wall bracket for RPC-4224, RPC-4220, RPC-4216, RPC-4116, RPC-4020, RPC-4164 | $11
3x Cooler Master JetFlo | 120mm | 95CFM/1200-2000RPM | 36dbA | $36
2x Nexus SP802512H-03PWM | 80mm | 38CFM/2800RPM | 16dbA | $20
////////// Decided against the NORCO case from all the bad things I've read about (it's just too many horror stories and I have bad luck)
SuperMicro CSE-836BA-R920B 3U | 920W Redundant 80PLUS Platinum | 5x 80mm fans (Should I switch out these fans or are they good enough?) | 16x SAS/SATA | $880
* Backplane: 3U Direct Attached Backplane, features:
3Gb/6Gb support
SES-2 Enclosure Management Support
SAS/SATA support
Four SFF 8087 connectors
Mini-I-Pass (SFF 8087) connectivity

* PSU:
Included in case. But will there be enough molex to connect everything? Is it modular? How can I make sure I won't run into problems here to connect with the motherboard?

* Mobo:
X9SRH-7F-O | $430
(As recommended by cyberjock, since I might want to expand in the future with 4-6TB drives and ZFS will need more than 32GB RAM.)
# Requires Revision K chassis, and my 836BA-R920B isn't listed under the "optimized" ones.. am I screwed? An optimized chassis would be 836E16-R500B which costs a few hundred more and only comes with a 500w PSU, what's going on here? This is my biggest compatibility concern)

* Memory:
2x 8GB Samsung M393B1K70DH0-CK008/Q | DDR3-1600 | CAS 11 | 1.5V | Registered ECC | $112

* CPU:
Xeon E5-1620v2 | HyperThreading | $275
(No CPU Cooling? So what kind of fan should I go for here? I have some stock intel i5 fan, would that be enough?)

* SAS HBA cards PCI x8:
1x SAS9211-8I 6Gbps | $200 (X9SRH-7F has onboard 8 port sff8087)
4x Monoprice SFF-8087 male-to-male cables | 36pin | $40
(I could just buy 1x 9201-16I $335, but since I already have onboard 2x sff8087 I kind of want to keep the drives separated on a hardware level too, which makes me feel safer since they won't share a SPOF. Is there an issue with the onboard LSI 2308 (firmware)?)

* Storage Disks:
16x2TB WD Reds = $1260 | 16x2TB x 1 RAIDZ-3 pools => 26TB usable
(Is there a better ZFS configuration I'm not thinking of here? If I want to change them up to 4TB drives down the road, is it dangerous to create such a large RAID (I suppose if I did a 2x8 raidz2 I'd lose usable storage but I assume rebuilding wouldn't be as dangerous since it's not such a large array)?)

* OS Disk:
SSD 840 EVO 120GB | $82
(I'd like to throw all the zfs metadata on here as well, so I end up with more storage on my zpool, does that make sense? And I'm wondering as well if I can just plug this SSD into the motherboard's SATA or would it have issues conflicting somehow with the other drives being connected to the LSI HBA cards?)

Total price tag: $3280

Before I'm going to spend that much money, I'd love to hear from the community here. I read through some guides and other first build stories, so I think I've created a scalable server that's going to keep me happy for many, many years with a good amount of expansion down the road.

For anyone taking a look at this, you have my gratitude.

 

[Ericloewe]

Is there an issue with the onboard LSI 2308 (firmware)?)

Not at all.

(No CPU Cooling? So what kind of fan should I go for here? I have some stock intel i5 fan, would that be enough?)

Depending on the exact layout, Supermicro has a few coolers available. You'll probably want an active one.

16x2TB WD Reds = $1260 | 16x2TB x 1 RAIDZ-3 pools => 26TB usable
(Is there a better ZFS configuration I'm not thinking of here? If I want to change them up to 4TB drives down the road, is it dangerous to create such a large RAID (I suppose if I did a 2x8 raidz2 I'd lose usable storage but I assume rebuilding wouldn't be as dangerous since it's not such a large array)?)

16 drives is too wide. 2x RAIDZ2 with 8 drives is better.

(I'd like to throw all the zfs metadata on here as well, so I end up with more storage on my zpool, does that make sense?

Things do not work that way. At all.

And I'm wondering as well if I can just plug this SSD into the motherboard's SATA or would it have issues conflicting somehow with the other drives being connected to the LSI HBA cards?

Connect it however you please, it won't make a difference.

 

[DearestDreamer]

I'm really excited to build this system,

Depending on the exact layout, Supermicro has a few coolers available. You'll probably want an active one.

Thank you for elaborating on that.

16 drives is too wide. 2x RAIDZ2 with 8 drives is better.

After I posted I actually was thinking that might be better yes.. thank you for confirming this.

Connect it however you please, it won't make a difference.

Alright that's good to know, I'll read more about optimizing the ZFS storage, but looks like I'll just end up buying two small SATA DOM (~20GB) and mirror them for now instead of the EVO.

I read that I might need a heatstink retention bracket since the case doesn't come with one, is that true? I thought the CPU/motherboard would have to take care of that.. I guess I'll find out once it's here though and buy one if I need to.

 

[Ericloewe]

Alright that's good to know, I'll read more about optimizing the ZFS storage, but looks like I'll just end up buying two small SATA DOM (~20GB) and mirror them for now instead of the EVO.

I read that I might need a heatstink retention bracket since the case doesn't come with one, is that true? I thought the CPU/motherboard would have to take care of that.. I guess I'll find out once it's here though and buy one if I need to.

Any required brackets are the responsibility of whoever provides the heatsink on LGA115x platforms. LGA2011 platforms include a bracket.

 

[DearestDreamer]

I tweaked the build a little and I'll be purchasing the case and motherboard separately first, to make sure it'll fit (Although SM tech support said it should, I'd still like to be sure before I order the rest). Mainly I looked through the OS compatibilities and I just felt the X10 has several PCIe slots that will come in handy. So this is it, I'm actually much happier about this and I'll let you guys know how it turns out once everything is here and assembled. If you notice something now please do tell, I won't be buying the rest until I test the motherboard with the case, which will definitely take a while. Thank you.

* Case:
SuperMicro CSE-836BA-R920B 3U | 920W Redundant 80PLUS Platinum | 16x SAS/SATA | $880

* Mobo:
SUPERMICRO MBD-X10SRL-F Server Motherboard LGA 2011 R3 | $275

* CPU:
Xeon E5-1620v3 | HyperThreading | $307

* Memory:
2x 16GB M393A2G40DB0-CPB | DDR3-2133 | CAS 15 | 1.2V | Registered ECC | $230

* CPU cooler:
SNK-P0048AP4 HSF | narrow ILM | $40

* SFF8087 HBA
2x IBM m1015 | PCIe x8 | Full Height & Length | $200

* SFF8087 cables
4x Monoprice SFF-8087 male-to-male cables | 1meter | 36pin | $40

* Storage Disks:
16x2TB WD Reds | ZFS 8x2 RAIDZ2 | $1260

* OS Disks:
2xSuperMicro SSD-DM016-PHI (comes with 5v cable) | $99

 

[nightshade00013]

Honestly I would grab a couple cheap ssd's since the price has dropped down so low vs the SATA DOM's.

Even though FreeNAS will not use that much space it will be cheaper and last longer since the wear leveling will allow more areas to be written than the 16GB DOM's Plus the case has space for three internal 2.5" drives to be mounted. It's a little savings and space for the future, FreeNAS could get a lot bigger later on when we get to version 11 and 12. A pair of 60 SSD's can be had for around 80 just have to find a couple sata cables. I know you were planning for a single larger SSD and then leaned towards DOM's and I am steering you back to SSD, everything has a tradeoff but since the case has the mounts why not use them.

Otherwise it looks good, but with what you are spending already may as well step up to 3 or 4tb drives. Could do a single 8 drive raidZ2 for less and then just add a second one later using 4TB drives and have the same space available. Eight 4TB drives would be around $1120 right now. Later on down the road they will be cheaper and you will end up with twice the space you are planning for.

That would be my only two changes to the build for now. But as it sits it should work fine.

 

[DearestDreamer]

Honestly I would grab a couple cheap ssd's since the price has dropped down so low vs the SATA DOM's.

Even though FreeNAS will not use that much space it will be cheaper and last longer since the wear leveling will allow more areas to be written than the 16GB DOM's Plus the case has space for three internal 2.5" drives to be mounted. It's a little savings and space for the future, FreeNAS could get a lot bigger later on when we get to version 11 and 12. A pair of 60 SSD's can be had for around 80 just have to find a couple sata cables. I know you were planning for a single larger SSD and then leaned towards DOM's and I am steering you back to SSD, everything has a tradeoff but since the case has the mounts why not use them.

Otherwise it looks good, but with what you are spending already may as well step up to 3 or 4tb drives. Could do a single 8 drive raidZ2 for less and then just add a second one later using 4TB drives and have the same space available. Eight 4TB drives would be around $1120 right now. Later on down the road they will be cheaper and you will end up with twice the space you are planning for.

That would be my only two changes to the build for now. But as it sits it should work fine.

Thank you for your comments! I'll look into the SSD prices and see if I can find two reliable ones instead of the SATADOMs, they just looked so cute when I discovered them and they fit nicely with my motherboard, lol. You make a valid point about the 4TB drives instead of the 2TB, because I do end up with the same amount of usable storage, 24TB, in either scenario and it won't be such a hassle to expand just another RAIDZ2 vdev rather than turning one of the 2TB ones into a 4TB one.. alright yeah it makes more sense. Thanks for your suggestions, it's really helpful when another voice rather than your own states things clearly. Thanks!

I also just realized that I'll probably need a rack cabinet for this.. is there a recommended one that's rather small (4-6U since I'll probably only be putthing this server in there and nothing else)? I can't find anything on newegg, ebay or amazon. The supermicro chassis has the following dimensions:
3U:
Height 5.2" (132mm)
Width 17.2" (437mm)
Depth 25.5" (648mm)

 

[nightshade00013]

LOL, no problem. Glad it at least made sense.

A rack cabinet would be good to have but I think your going to have a lot wrapped up into it cash wise. Most of the small ones are running 450mm deep unless you step into the 600 range. If you have a closet or an area where there will not be a ton of dust I would set a desk for it to be up off the floor. My father has a 1U rack that he is going to mount on a wall sideways (top facing out and drives on the right ports on the left) so that could also be a possibility if you have a place that would work well for it and can find or make some brackets to do it. The things to worry about the most are securing it against damage or accidental movement and dust infiltration.

 

[nightshade00013]

Was just looking and BestBuy has some 120GB SanDisk SSD's for 40 so that would be something to jump on.

http://www.bestbuy.com/site/promo/sandisk-ssd-1-day

 

[Ericloewe]

DOMs are cute little things, but they're also crazy expensive. I'd definitely go with real SSDs, even if it complicates cabling somewhat.

 

[DearestDreamer]

Hmm, I have a question about the OS disks, in particular with ZFS. (I read about this here https://clinta.github.io/FreeNAS-Multipurpose-SSD/)

If I don't mirror them, will I be able to go with this setup:
L2ARC and ZIL on one 120GB SSD (one with supercapacitors).
FreeNAS/OS on another ~40GB SSD.

Or is it possible to put L2ARC,ZIL and the OS on one single SSD? If I split it in two, ZIL/L2ARC and OS, is this considered bad practice? I read it's possible to keep ZIL/L2ARC on the same SSD but I just wanted to ask some more experienced users if there's something bad about that kind of setup (I realize there might be a performance trade-off, but I think that's okay with how fast SSDs are).

 

[nightshade00013]

Worry first about the boot drives and go from there. The likelihood of ZIL or L2ARC being of much good in a media server is pretty low. Plus ZIL needs to be mirrored if you are using SSD's, on top of that most people aim for SLC SSD's or ram drives.

It all sounds nice and fun but unless you are going to be hammering the FreeNAS with many users and lots of IO they are not going to give you much if any benefit. You are much better off just mirroring your boot drives.

 

[DearestDreamer]

Thanks nightshade, that sounds the same to what I've read about it.. I have a powerful CPU, though so I was wondering if I might do some video editing, if ZIL and L2ARC would prove useful.

I'm just a little concerned about the space, since I only have 2 2.5" slots, in which case I'd rather use that for ZIL/L2ARC and put the OS on two mirrored USB sticks, wouldn't that make more sense (now considering video editing might occur)?

 

[nightshade00013]

Even if you do some video editing unless your going to be moving major files by multiple people zil and l2arc will be of little help. I transcode files from one format to another across the network to my FreeNAS and there is little impact. You will probably just be wasting money and hurting your performance by adding them.

 

[DearestDreamer]

I see, thanks for reassuring me on this and I'm definitely not going to have multiple people access my server, at most 2 devices with minimal impact. I'll read more about zil and l2arc then, but from the way you've described it, it really doesn't seem like my use case calls for them, so I can just put my OS on those SSDs without worrying about zil and l2arc. I assume even if I wanted to use them on SSD devices later on, I could just execute those commands and enable zil/l2arc on SSDs..

Thanks! I'm so excited for getting all the parts together, I already got the drives and the motherboard, I'll have to make sure I test each component before I start using the server properly. I'm glad there's some good tutorials and guides on testing in the forums.

EDIT: So while most components have arrived, I'm a little sad that the chassis is not new as it was advertised on Amazon. It appears to be in good condition but it's clearly been used before (last company has stickers on it). So that's a bit of a bummer, but I don't think pursuing this further with the seller will yield anything positive (Like a rebate or something..).

What I also noticed is that I don't seem to have any screws to install the motherboard, the chassis just came with the bare minimum basically and the motherboard's contents were just these:

I thought the case usually shipped the necessary screws, but I guess this time I got screwed on that :)

 

[DearestDreamer]

So I got it all together! Awesome, although there's a few things I'm not quite sure about, since it's my first Supermicro case + motherboard. If anyone has an idea on what to do here, I'd be really glad.

Basically on the first on here you can see how I attached the cables from the PSU to my motherboard, but there's one cable that's still unplugged and I don't know where to connect it to?

I'm also really confused about the way the front panel connects to the motherboard, I'm familiar with multiple small cables, but I've never dealt with a huge one like this that doesn't even properly have the labels on it, so if anyone could comment on whether I did this right, I'd feel much more relieved (I seem to lack the NMI connection as you can in the picture.. is this normal?).

Other than that, everything fit properly. I'm happy the CPU Fan was small enough for the air shroud to fit on top.. do you guys think it's fine if it touches it a little? The air shroud seems to be made out of plastic, so I'm not quite sure if this is how you're supposed to do it, or if you should remove the air shroud altogether if you have active cooling. I also didn't know I had a chassis interruption connector but I guess that's kinda useful.

Here's one last image with almost everything connected:

 

[Ericloewe]

Basically on the first on here you can see how I attached the cables from the PSU to my motherboard, but there's one cable that's still unplugged and I don't know where to connect it to?

Given the connector, it's either ATX12V for a second CPU or PCI-e for graphics cards. Given that it's labeled "motherboard only", it's the former.

I'm also really confused about the way the front panel connects to the motherboard, I'm familiar with multiple small cables, but I've never dealt with a huge one like this that doesn't even properly have the labels on it, so if anyone could comment on whether I did this right, I'd feel much more relieved (I seem to lack the NMI connection as you can in the picture.. is this normal?).

It's normal for Supermicro to have everything setup to minimize effort during installs. If you followed the manual, there should be no problem.

 

[DearestDreamer]

Alright thanks Eric! I figured the extra power cable might've been for a PCI extension, but it makes more sense that it's there for multiprocessor boards.

I booted the system and it seems to work fine so far, I'll start testing the RAM and the other components and hope everything's fine. But so far it all seems functional and great! The server is honestly pretty quiet, I didn't expect that.

Here's my temps without the rear fans (waiting on extension cables...) for anyone who's interested. Looking pretty good I think for the speed the fans are running at (really quiet).

 

[DearestDreamer]

So I've set out to get my two LSI 9211-8i into the proper P20 IT firmware and I'm not sure if one of them (v5.00.13, slot 00) might already be messed up?

Looks like different outputs give me different information on whether it's already in IT or IR firmware. Which do I believe? Is the card bad?

 

[jgreco]

Plus ZIL needs to be mirrored if you are using SSD's, on top of that most people aim for SLC SSD's or ram drives.

Nothing in this sentence is true in 2015.