Validation - 1st FreeNAS Medium Build

[GrumpyBear]

Hello,

and thanks to the community here. I've been lurking for a while and have learned lots.

I currently am using a Synology DS212 as a backup and media server (Plex). This unit is used to backup a 2TB RAID 10 array (4 x Hitachi 1TB DeskStars) on my workstation which is a older SuperMicro box running WIN7Pro 64. I do mostly Photo Editing (PhotoShop CS) and am starting to run out of Storage on the RAID 10.

After some investigation decided to do a FreeNAS Build instead of getting a larger Synology box as I can replace the Mobo in my workstation and re-purpose the old one for a FreeNAS build.

I will move all my storage off the Workstation onto the NAS and the Synology will be used now as the on-site backup for the important stuff (I off-site using a pair of USB3 enclosures).

Plex will also be moved to the FreeNAS. I don't torrent and usually just the the Plex for ripped DVDs and HomeVideo so the Plex is currently NOT transcoding but I may want to transcode one or two streams in the future. I may also look at installing a MineCraft server for the kids at some point in the future.

Here's the hardware I've chosen based on what's on-hand and the "Hardware Recommendations" sticky I already have the NAS drives and the Case and PS are on their way. Feedback on the hardware and any suggestions or potential "Gotchas" would be appreciated.

1 x Fractal Design R4 Case
1 x Corsair RM550 550W Modular ATX12V Power Supply
1 x SuperMicro X9SCL Motherboard (re-purposed) 6 x SATA2 3Gbps
1 x Intel Core2 i3-2100 3.1 GHz CPU (re-purposed)
1 x Kingston 8GB (2 x 4) DDR3 PC10600 CL9 ECC Unbuffered RAM (re-purposed)
3 x Western Digital WD30EFRX 3TB 64MB cache HDD
3 x Seagate ST3000VN000 3TB 64MB cache HDD​

Thermal Concerns:

Will the one front 140mm fan in the case be sufficient. It looks like it covers the top 5 bays and as heat rises should also help the lower 3 bays?
Load the bays top-down?
Any reasons to not use the PWM Headers on the MoBo and use integrated Fan Controller instead (which is pretty flimsy)?​

Power Concerns:

Will 550W be enough headroom and room for a little growth (say 2 - 5 more disks) ?
12V Startup Disk load 11.4A (136.8W) :

Seagate NAS Disks: 3 x 2.0A = 6.0A
WD NAS Disks: 3 x 1.8A = 5.4A​

Should I bother stagger starting the disks?

Quiescent Load (< 200W):

Seagate NAS Disks: 3 x 4.8W = 14.4W
WD NAS Disks: 3 x 4.1 = 12.6W
CPU TDP: 65W
Fans 10W?
Mobo 30W?
​

The Grumpy One

 

[danb35]

It should work, though you may find you want more RAM. If you were buying a new motherboard, I'd recommend one of the -F models to get IPMI. That's a very handy feature, but I don't know that it's handy enough to justify throwing out perfectly decent hardware. If you were buying new RAM, I'd recommend 8 GB sticks rather than 4 GB, since your motherboard only has 4 DIMM sockets, and more RAM is usually better for FreeNAS (particularly if you're using Plex). But again, you have them, so you might as well use them. You could still fit up to 24 GB into your board with them there, which should be adequate.

Corsair power supplies and Kingston RAM aren't held in very high regard around here; opinion seems to favor Seasonic and Crucial, respectively--but since you've already ordered the PS and you already have the RAM, I'd figure go with what you have. Your PS capacity should be ample--my build (which you see in my signature) uses the same basic motherboard, a Xeon E3 CPU, and 12 disks, and I've never seen it draw over 200 watts from the wall.

 

[GrumpyBear]

Thanks Dan. I figured I would eventually be buying more RAM if I start transcoding with Plex.

BTW my new Mobo for my workstation will have IPMI and I'm getting Crucial ECC for it once I decide on the E3-1231V3 or the E3-1241V3 or ...

 

[marbus90]

O joy! Someone wants to repurpose his old computer. Which has ECC. Thats rarely seen, let's hide away and watch this rare specimen... :D

The chosen PSU is good for up to 12 regular HDDs plus 6 more WD Red up to 6TB. The WD non-Pro Reds are afaik the only disks which don't need the 2A@5V during spinup, therefore you can add some more of them.
All other components are somewhat close to community choices.

For the workstation I'd decide on the 1241 v3. You may never know when you want that little bit more singlethreaded speed.

 

[Ericloewe]

Power is fine, but the Seasonic G-Series is a bit better than Corsair RM - Seasonic only uses Japanese capacitors, for instance, while Corsair has some second-tier stuff like CapXon. A G-450 would be enough for more than 10 HDDs.

Let the motherboard control the fans. That's my opinion at least, but I really dislike dumb fan "controllers".

 

[GrumpyBear]

O joy! Someone wants to repurpose his old computer. Which has ECC. Thats rarely seen, let's hide away and watch this rare specimen... :D.

LOL I've been buying SuperMicro at work for years and they were a natural for my home system as I wanted something rock solid and was willing to trade off some performance and features at the same price point. The current system has been running almost 3 years 24x7 with no issues.

Thanks for the advice on the power supply Eric but It's on order so I'll have to think about if I want to eat the restocking charge.

I was leaning towards not using the fan controller as well as I'd rather be able to see and control the fans individually. I did note that the Fractal fans are both 3 wire which is a little weird these days so I might replace them with 4-wire versions.

The Mobo in my workstation with slightly fewer drives runs at 30c on the CPU with minimal load and 56C about ten minutes into a 1080p MP4 render so I suspect the 140mm fans in this case ( as opposed to the 90mm current ones) should give as good, if not better airflow.

 

[GrumpyBear]

Went to see if the case and power supply were delivered last night and they are still in transit. So I took the opportunity to change from the Corsair to a SeaSonic Power Supply as recommended knowing that 550W likely greatly exceeds the current capacity requirement but will give me some room to grow.

I have done some soul searching over the last little while and have been researching FreeNAS more here and by RTFM. Though I was looking at re-purposing my I've decided to go with new hardware for the build as staring with 8GB leaves no RAM for accommodating the 1GB RAM per 1TB capacity rule of thumb and adding more DDR3-1200 just seems a waste. I've also decided that Kingston just is too risky and to go with Crucial for the ECC RAM.

So I'm going with:

Case: FractalDesign Define R4 Black Pearl Window
Power Supply: SeaSonic G Series SSR-550RM 550W
Motherboard: SuperMicro X10SL7-F-O
Processor: Intel Xeon E3-1231 v3 (8M Cache, 3.40 GHz)
Memory: Crucial 16GB kit (8GBx2) DDR3 PC3-12800 Unbuffered ECC 1.35V 1024Meg x 72 CT2KIT102472BD160B
HDD: 4 x Western Digital Red WD30EFRX 3TB & 4 x Seagate NAS ST3000VN000 3TB​

An will likely max the memory out fairly soon as well.

Thanks again for the input. It is important to me that this box be rock solid and have some room to grow so I decided to go with SuperMicro board with the integrated SAS controller rather than one of the highly recommended IBM-branded LSI HBAs as this will give me a couple of PCIe slots for expansion.

Once I get the hardware and start the build I post some pics and design decisions with respect to cooling etc and some power measurements from my Kill-a-Watt and UPS in a separate thread.

I've already decided not to virtualize (it's a prod box) and not to run iSCSI as the lack of flexibility to me outweighs any potential performance benefits.

Off to learn more about zpools, vDevs and other arcane things specific to FreeNAS and FreeBSD ...

 

[GrumpyBear]

Grrr.

Substituted a Corsair HX650 Power Supply after my favorite local vendor still hadn't received the SeaSonic when the Fractal Define R4 case was delivered. It has "Quality Capacitors" whatever that means.

The Motherboard, RAM, fans and Evercool 5-way, 4-pin fan splitter arrived but the Xenon E3-1231 is on a slow boat and is somewhere between Vancouver and here.

Mounted the components anyway (Build Pic here ). Noticed that the X10SL7-F does not quite line up with the I/O shield in the Fractal Design R4 case. The board is about 2mm too low. I noticed others in this cases' reviews have experienced this as well. The fix was to add a couple of flat washers under the mounting posts at the rear of the board.

Plugged the unit in and attached a Ethernet cable to the IPMI interface. After a minute or so noticed the BMC heartbeat and saw a lease obtained on my router. I am able to successfully connect to the IPMI but can't do much with it as even the sensors aren't working without the CPU. Probably has something to do with the BMC having to go through the PCH to get to the Nuvoton NCT6776D I/O Controller which has the sensors attached.

Downloading the CD for the Motherboard. Why SuperMicro does not include this is a bit of a mystery.

I have downloaded the v16 IT firmware for the on-board LSI 2308 and will be flashing it once I have a processor ...

 

[GrumpyBear]

Further thoughts as the disks are being torture tested (thank-you mr grinch for convincing me of the need to frustrate my tinkering urges for hundreds of hours).

I have a love hate relationship with the size of the fractal design R4. What sold me on this case was the cable management and clean discreet appearance with good options for fan placement. This beast is, in my opinion, unnecessarily large for a smallish NAS build. It is, after all, designed with gaming in mind and while that leads to it having good fan placement the width of the case seems to be thee to accommodate big CPU coolers and exotic water cooling systems that aren't required here.

If I had to do this again I'd probably go with a narrower mid tower case and stuff 6 4TB disks in it rather than 8 3TB disks. With RAIDZ2 that would mean 16TB versus 18TB. I might even consider the define mini though it is still fat it still has that certain sleek asthetic look I find appealing. I waver between an atom based ITX build and went with microATX variant as it gave me more options.

With respect to the motherboard the X1oSL7 is overkill for just a 6 disk build and the X10SLL or M would have saved some funds. But it does give me some options. I was thinking it could allow me to add a JBOD enclosure with a SFF8087 for expansion but if I wanted to go big I'd likely just knuckle down and fork out the cash for a SuperMicro 24-bay chassis on eBay. In all likelyhood if I do need to upgrade capacity in a few years I will probably grow the pool by slowly replacing the disks with whatever is the best bang for my buck at that time.

I'm still mulling over 8 disk RAIDZ2 for 18TB versus 8 disk RAIDZ3 for 15TB (really closer to 15TB versus 12TB when you do the math with real numbers and take into account Swap). I suspect that if I can come close to saturating GigE with Z3 (I know I can with Z2 - 108-118MBps measured in test with some large sequential files) I will probably use that.

With respect to the cooling options...

I picked up 4 120mm Cougar fans to replace the 2 140mm Fractal fans that came with the R4. The thinking was that the Cougars have PWM and a rated minimum speed of 800rpm with lower noise so they should be able to move the same amount of air and keep the drives cool (>40C under heavy CPU and disk utilization with a 25c ambient maximum). In practice, however, I'm not too happy. Under the extended sequential read tests (which seem to be hotter than the write tests) two of the disks are at 40 and 41C respectively. The main issue seems to be that the Cougar Fans are way out of spec and IPMIutil reports between 485 and 550 rpm at minimum (30% modulation) rather than the 800rpm claimed. This could be due to an unpublished tolerance in the specs or back pressure effects. I have 3 Noctua 140mm NF-A14 3000 rpm fans on order and a 120mm one as well. Why the 120mm one? Because NewEgg only had 3 of the 140mm fans in stock also I'll see if running the 120mm fan in the case back might force more air out the expansion card area. The Noctuas are spec'd at 800 rpm minimum as well but being 140mm should move more air. Deltas were briefly considered but they are way to noisy.

 

[marbus90]

saturating 1gbe is no problem. every proper freenas system does that. it doesn't matter which disk layout you pick. it's never a problem.