BUILD Please review my Intel Skylake FreeNAS build

[snicke]

After a lot of reading, googling, asking and thinking it's time for me to start buying the parts for my brand new Intel Skylake based FreeNAS server. But before that, of course I would like all the skilled and helpful people of the excellent FreeNAS community to review my build. :)

The use case:

• Most important: Secure storage for my family pictures, videos and files.
  • I really want a system where the probability of data loss is so unlikely that "it will not happen in practice" except if my server explodes/house burns down/theft etc. I always have two backups of my important data, one on site and one off site, so I don't at all look at it as a backup replacement but I do think that if you have chosen a system like FreeNAS mainly for data integrity purposes your backups should only come into play in an unlikely emergency situation where the RAIDZ configuration really didn't matter.
• Media server for movies, music, TV-shows etc. Using Plex when needed. Though probably using Kodi on the HTPC.
  
• Owncloud: Be able to access all my files remotely in a convenient way.
• Jails: At least one Redmine and one Wordpress server.
• GIT-repos.
• Enough head room to be considered future proof both regarding hardware, expansion possibilities and initial storage capacity and vdev configuration.

The build:

• Motherboard: Supermicro X11SSM-F
  • X11SSH-F as an alternative and then getting an M.2 PCIe 3.0 (X2 speed!) port but losing an ordinary PCIe 3.0 slot (X4 speed!). Feel free to give me your opinion of what is most useful.
• CPU: Intel Xeon E3-1240v5 3,5GHz Update: Intel Xeon E3-1230v5 3,4GHz since I cannot find 1240 in stock.
  • Yes, why not E3-1230v5 @ 3,4GHz? So little difference in clock frequency/speed! Well, the local price difference is so low so why not 3,5GHz with the 1240.
• RAM: 2x Samsung 16GB DDR4 ECC 2133Mhz UDIMM (M391A2K43BB1-CPB - unbufferd) for a total of 32GB.
• Storage: 8x WD Red 4TB drives (WD40EFRX) in RAIDZ3
  • Main motivation of RAIDZ3: See "Most important" in use case above.
  • Usable storage in RAIDZ3: 5*4*0,8 = 16TB.
  • (Relevant comment from @Ericloewe: That's 16 drive-manufacturer-TB, not 16*2^40 Bytes. You get closer to 14.5TB.)
• PSU: Corsair RM650i 650W
  • 86,16% efficiency @ 10% load @ 230V (source: Corsair)
  • 90,34% efficiency @ 20% load @ 230V (source: Corsair)
• Boot SSD: Samsung 850 EVO Series MZ75E120B 120GB
  • Yes, overkill for boot drive both size and speed wise, but the local price difference for all smaller and slower drives (branded and "no names") is so small so why not go with a really good and probably reliable boot drive. Cannot use USB stick as boot drive (and probably don't want to = follow current community the trend of using SSD) since FreeNAS currently doesn't support XHCI for USB which makes the USB ports on the Supermicro unusable until XHCI for USB becomes supported by FreeNAS in the future.
• HBA: IBM ServeRAID M1015
  • For additional SATA ports at least for the boot SSD but maybe also for a jails SSD and future expansion. The motherboard has 8 Sata3 ports though for the planned 8 storage drives.
• Update: 1x (or 2x in mirror) Samsung 850 EVO Series MZ-75E250B 250GB SSD pool for jails specifically.
• Chassi: Fractal Design Define R5 OR Fractal Design Node 804 OR The best thing that the skilled FreeNAS community can come up with.
  • Please come up with your proposals and why! Obviously a not to pricey and silent chassi is wanted but where the highest priority is that all the computer parts are kept cool enough and well ventilated.

An bonus question: There are 2x 1Gb Ethernet LAN ports on the X11 motherboards considered above. However, there are Supermicro X11 boards with 4x 1Gb Ethernet LAN ports aswell. To me it sounds totally overkill/useless with more than 2 LAN ports but feel free to comment if you have any other opinion and why.

Looking forward to your input fellow FreeNAS geeks! Thank you in advance! :)

 

[Ericloewe]

Usable storage in RAIDZ3: 5*4*0,8 = 16TB.

Careful, that's 16 drive-manufacturer-TB, not 16*2^40 Bytes.

You get closer to 14.5TB.

An bonus question: There are 2x 1Gb Ethernet LAN ports on the X11 motherboards considered above. However, there are Supermicro X11 boards with 4x 1Gb Ethernet LAN ports aswell. To me it sounds totally overkill/useless with more than 2 LAN ports but feel free to comment if you have any other opinion and why.

You'll probably never use the second port, never mind the third or fourth.

The X11SSH-LN4F uses the two missing PCI-e 3.0 lanes to feed two additional Intel i210 NICs. Maybe if you plan to retire your server into a router role or something... Otherwise, the X11SSM-F tends to make more sense in most scenarios.

 

[snicke]

Careful, that's 16 drive-manufacturer-TB, not 16*2^40 Bytes.

You get closer to 14.5TB.

You're totally right. Thank you. Will add your calculations above.

Still currently enough for me though.


Sent from my Galaxy S6 Edge

 

[ChriZ]

I would probably choose to go for 6x6TB in Raidz2
Less drives, less heat, less power consumption, more available space, ideal drive count for a very safe raidz2 . And since you will have two spare sata ports, there will be no need to spend money on extra HBAs.
Just my 2c.

 

[Arwen]

I would probably choose to go for 6x6TB in Raidz2
...

I'd probably go with the above. Or even 6 x 5TB in RAID-Z2, which is about the same amount of storage
as a 8 x 4TB in RAID-Z3. With 2 parity drives, (RAID-Z2), you get less overhead for writes than 3 parity
drives, (RAID-Z3).

If you are paranoid, you can get 7 drives, and keep one as a spare, (cold, warm, or hot).

Spare definition:
Cold

Kept in a box, awaiting install.
If un-tested, could be DOA :-(.
Even if tested, still could suffer infant mortality syndrome.​

Warm

Installed and spinning.
Requires manual assignment or configuration when needed.
Could be used for temp storage, backups, or anything else, until needed as a replacement drive.​

Hot

Will automatically take over from any failed drive for a pool.
Unable to be used for any other purpose.​

 

[Dice]

If you are paranoid, you can get 7 drives, and keep one as a spare, (cold, warm, or hot).

Sorry if this is FAQ: Are HOT spares supported by FreeNAS?

 

[Ericloewe]

Sorry if this is FAQ: Are HOT spares supported by FreeNAS?

Yes, it's a recent feature.

 

[snicke]

I would probably choose to go for 6x6TB in Raidz2
Less drives, less heat, less power consumption

Shouldn't be a huge diffence though, remember that each 6TB drive has higher power consumption compared to each 4TB.

Found the following numbers:

• Idle power for 4TB: 3,3W
• Idle power for 6TB: 3,4W

6x 6TB will draw 0.1*6 = 0,6W more in power compared to 6x 4TB. The 2x extra 4TB will draw 2x 3.3W = 6.6W more. In total 6.6 - 0.6 = 6W more power for the 4TB solution @ idle adding to heat and consumption. Quite acceptable.

But even less difference for the worst case while in seek:

• Seek power for 4TB: 4,5W
• Seek power for 6TB: 5,3W

6x 6TB will draw 0.8*6 = 4.8W more in power compared to 6x 4TB. The 2x extra 4TB will draw 2x 4.5W = 9W more. In total 9 - 4.8 = 4.2W more power for the 4TB solution @ seek adding to heat and consumption.

Thanks for your input though. However, I still leaning towards having 8x 4TB drives in RAIDZ3 for the reasons in the first point of the use case in OP. I probably add an 256 GB SSD (or 2 in a mirror) for jails though.

More thoughts regarding my build anyone?

 

[jgreco]

Yes, it's a recent feature.

What's your definition of "recent"? Because it's been working (admittedly perhaps not reliably) ~~since 9.3 came out; I had a disk replaced automatically about a year ago. Very pleasant surprise. I had set it up as a spare just to keep track of it.

 

[Ericloewe]

What's your definition of "recent"? Because it's been working (admittedly perhaps not reliably) ~~since 9.3 came out; I had a disk replaced automatically about a year ago. Very pleasant surprise. I had set it up as a spare just to keep track of it.

"Recent" is what I call anything not introduced in a point release. I hope to discourage people from asking when it was introduced because I have no patience to try and figure out when it was. :p

 

[jgreco]

(... *not* introduced in a point release...? .... doesn't parse into anything sensible to my caffeine-lacking brain...)

Okay, so "recent", just so we're clear, is "for about half of @Ericloewe 's time on these forums."

 

[Ericloewe]

(... *not* introduced in a point release...? .... doesn't parse into anything sensible to my caffeine-lacking brain...)

Say, released between 9.3 and 9.3.1, during the long stretch of confusing "we're adding several features that would be worthy of note, but treating it as just another update. Also, bugfixes" updates.

 

[snicke]

@jgreco What's your opinion and thoughts regarding my build in the OP? Would really appreciate your feedback.

 

[jgreco]

Say, released between 9.3 and 9.3.1, during the long stretch of confusing "we're adding several features that would be worthy of note, but treating it as just another update. Also, bugfixes" updates.

Which seems to boil down to most of them, huh.

 

[jgreco]

@jgreco What's your opinion and thoughts regarding my build in the OP? Would really appreciate your feedback.

Yeah, no, not really. That's no guarantee that it carries a Grinch seal of approval, since a lot of the time I'm just looking for the way-obvious-stupid or the esoteric bits that almost no one else would know to comment on.

If I were to say anything, it'd be along the lines of "why Skylake", because I generally prefer not to be on the bleeding edge.

 

[gpsguy]

A couple of weeks ago they were going to bump the version to 9.3.2 with the next SU. Since then they changed it to 9.3-STABLE.

Say, released between 9.3 and 9.3.1, during the long stretch of confusing "we're adding several features that would be worthy of note, but treating it as just another update..

Sent from my phone

 

[jgreco]

So the version number's moved down...? Cool.

 

[jde]

I'm a big fan of the R5 case. I went with it for my recent build and could not be happier with it. Its quiet and cool. It's in a regular office environment and I don't ever see drive temps in excess of about 30 degrees. It comes with two 140 mm fans and I bought another one to match. Two fans in the front for intake and one in the back for exhaust. I removed the front door panel from the case for improved air flow. I considered the 804, but it looks to me like changing out a failed drive in that case would be a PITA.

 

[religiouslyconfused]

I have been looking at the R5 as well as the Node 804. R5 seems to be cheaper on Newegg and maybe I might go with that.

 

[jde]

IIRC Newegg puts the R5 on sale in the ~$70 range with some regularity.