Advice needed (FN 10) - Go for a 2011-3 based build or stick with 1151?

[freakinlaser]

Greetings!
With the pending release of FreeNAS 10 I am pondering the plunge from my current Synology solution to a FreeNAS 10 solution which is powerful enough to run some other services.
Intended use:
- Fileserver (both "throwaway" media files, and critical documents and "marital problems if it gets lost" photo archive - will be used by my dad and my sister remotely. Off-site backups will be sent to their houses to off-the-shelf Synology boxes)
- pfSense
- OpenVPN server
- Rsync destination for other family members elsewhere in the country
- Plex
- Domoticz
- Various VM's to do various things (to play around with, initially)

Now, I am just about ready to commit to a Raid-Z2 setup with 6x WD Red 6 Gb. I would furthermore like to stick to mATX in a Fractal Design 304 or possibly 804.

What I am still on the fence about is this:
Go for 2011-3 or 1151?

For the 2011-3 build:
Mainboard: ASRock Rack EPC612D4U (because it's a bit easier to find than the equivalent Supermicro option)
CPU: Start out with a Xeon E3-2620 v3
Memory: Start out with 32 GB - go to 64 GB if I need it

For the 1151 build:
Supermicro X11SSL-F or GA-X150M-PRO ECC (because of the M.2 slot)
CPU: Start out with a Xeon E3-1240 v5
Memory: Start out with 32 GB- go to 64 GB if I need it

The 2011-3 build will give me more headroom to upgrade the CPU later on, if I do end up doing more with virtualization, but the question is:
- Will I need it for the intended use?
- Will upgrading be a smart choice at all, or should I instead opt to build a dedicated virtualization application server if the Xeon E3-1240 v5 turns out to be underpowered?
- Or should I go one step further, put a Skylake Pentium chip in my FreeNAS build, and save my money for a later-built application server altogether?

I am leaning towards the second choice - building with a Xeon E3-1240 now, and if I run out of headroom build something new to play with and keep the Xeon as my file server.

Input, thoughts?

Thank you in advance!

 

[JohnDigital]

ASRock Rack has been questionable lately search around and you will see. Do us all a favor and stick with tried and true, the newest Supermicro and the best Xeon you can afford. It can handle all of the tasks you have highlighted just fine, with the exception of PFSense. Nobody here will be in favor of you running your routing and firewall in the same machine as your server. I use an old Lenovo T61 Laptop (even T41's will work for this purpose, and they are cheap AF) and a Netgear PMCIA Gigabit card for mine. I was skeptical of the Netgear card, because it uses a Realtek chip, but it has been GREAT for the WAN side of the network and it IS gigabit. It works so beautifully im ashamed I haven't been doing that since forever. It draws a handful of watts and has a built in UPS (battery) and monitor. You can run OpenVPN on it too!

I havent run FreeNAS 10 at all yet because I am so comfortable with jails and CLI. Im not sure I will ever run it. I have everything running so perfectly im kind of sickened about the thought of being forced to switched to 10 at some point.

Anyways, I think you are on the right track, but I would go ahead and just start with 64gb ram, and be done with it. I dont see you needing to upgrade anything, anytime soon. One last thought, you should also start with a couple fairly large SSDs in at least a mirror for jails/containers/VMs. That would round out your machine and make us all jealous.

Those are my thoughts on the subject. Good luck, and welcome!

 

[freakinlaser]

Thank you for your response, John. The ASRock rack thing makes my choice a little easier, since the SuperMicro mATX Socket 2011-3 options all have a lead time disadvantage and I'm an impatient kind of guy.

ASRock Rack has been questionable lately search around and you will see. Do us all a favor and stick with tried and true, the newest Supermicro and the best Xeon you can afford.

OK, but the "best Xeon I can afford" option leaves the door wide open to go with something ridiculous - such as the 14-core E5-2683, since those can be found online in some pretty good deals occasionally.

It can handle all of the tasks you have highlighted just fine, with the exception of PFSense.

Anyways, I think you are on the right track, but I would go ahead and just start with 64gb ram, and be done with it. I dont see you needing to upgrade anything, anytime soon. One last thought, you should also start with a couple fairly large SSDs in at least a mirror for jails/containers/VMs. That would round out your machine and make us all jealous.

Actually, I was thinking about just one M.2 PCI-e SSD (Intel 600p series). I don't really need to make that redundant. Push comes to shove, I can live with a little bit of downtime for the system itself while I arrange for another boot device - if I leave pfSense off it, a 100% uptime guarantee is not needed.

For redundancy in the SSD I'd have to go for SATA, and possibly a larger case to make that comfortable... and I really would prefer to stick it all in the Fractal Design 304, keep everything nice and compact.

 

[JohnDigital]

OK, but the "best Xeon I can afford" option leaves the door wide open to go with something ridiculous - such as the 14-core E5-2683, since those can be found online in some pretty good deals occasionally.

Im all for ridiculous.

For redundancy in the SSD I'd have to go for SATA, and possibly a larger case to make that comfortable... and I really would prefer to stick it all in the Fractal Design 304, keep everything nice and compact.

Totally fine, just a matter of preference. My machine is in a basement, with no cares about size, noise, or aesthetics, but the Fractal XL, are very eye pleasing and quiet. There is no script, whatever your preference is, thats what makes FN such a smart choice.

 

[JohnDigital]

Define mini and arc midi are also viable options. They are very small and compact. I think the midi holds 8 drives and the mini 6, both have two 5.25 bays that you could put one of THESE in.

 

[freakinlaser]

Im all for ridiculous.

I'll mull it over a bit. If I buy RAM + CPU used, the total price difference on the system comes in at something like 10%, but chances are I'll have to take a risk on RAM that hasn't been tested by SuperMicro which would be rolling the dice.

 

[freakinlaser]

Alright, I decided to go with the 2011-3 build. Unfortunately I got outbid for a nice Xeon 2683 14-core, but I'll have the upgrade option if I ever need it.

Here's what I have coming in the next few days:
Mainboard: SuperMicro X10SRM-F
CPU: Xeon e5-2620 v3
RAM: 4x16GB DDR4 2133MHz ECC RDIMM Samsung M393A2G40DB0-CPB
Boot drive: Intel 600p 256 GB SSD (m.2 pcie x4)
PSU: Corsair CS550M
Case: Fractal Design Node 304
CPU fan: Noctua NH-D9DX i4 3U (supports narrow ILM, and I figured it might be better and quieter than a standard SuperMicro offering)

Now, I went off-reservation (off-qvl) with the RAM, but considering what I paid for it I consider it worth the risk - worst comes to worst I'll put it up for sale again.

Haven't bought any drives yet, I intend to play around with it first to familiarize myself with freenas before moving my data to it.

 

[JohnDigital]

Looks like a humdinger!! That drive is woefully too much to be booting FreeNAS from (still unsure if X10's even support boot from an M.2). Did you mean a jail drive? That thing will be sweet as a VM/jail/container storage.

Suffice to say I am jealous sir!

Edit: Hell yeah it supports NVMe boot!

 

[freakinlaser]

Looks like a humdinger!! That drive is woefully too much to be booting FreeNAS from (still unsure if X10's even support boot from an M.2). Did you mean a jail drive? That thing will be sweet as a VM/jail/container storage.

Suffice to say I am jealous sir!

Edit: Hell yeah it supports NVMe boot!

John, I was kind of assuming it could do both; so store the OS itself as well as some VMs / jails / anything other than my actual storage pool.

If it can't, I'll be booting FreeNAS from a USB stick.

Can't wait to start putting it all together. Next step will be to select some drives. I know I said I'd go with WD Reds, but I occasionally see some good deals on Seagate Enterprise disks, so I'm still a bit on the fence about that - but like I said, familiarize myself with the OS first, get comfortable with it, and then put in the drives and actually put it in service.

 

[JohnDigital]

No, sadly, it cant. USB is only slow(er) for booting and upgrades. I ran off of USB for a long time.. You are going to be fine, just fine!

Hell of a machine!

 

[Dice]

Nobody here will be in favor of you running your routing and firewall in the same machine as your server.

Don't be too sure about that... ;)
I'd advocate such a solution. I believe @joeschmuck runs this it all in one too. But we run ESXi as hypervisor.

A few precautions on the network security aspect:
- make sure a dedicated NIC is pass through to pFSense.
- make sure ESXi management network is on the internal side of the firewall, in the virtual network.

As for hardware @freakinlaser :
- 2011-3. Grants you virtually limitless RAM capacity. Which is great. 64GB is really not future proof by any means with that amount of functionality desired out the gate.
CPUwise. I'd go for a E5-1620, unless you've multiple VM's that should be capped or dedicated in terms of CPU cores. Letting capacity flow is ...sort of the cool part of virtualization.
- Stick with Supermicro, no further argumentation there.
-I'd bend the small case requirement a bit, since you're getting such powerful hardware it is ...sort of a waste to be locked out from expansions.

Good luck.

 

[Evi Vanoost]

You CAN run an all-in-one device IF you know what you're doing. The 'problem' obviously is your dependencies (what needs to run before the other things run).

If something goes wrong (eg. an update breaks stuff) and your server is dependent on the Internet to fix it but your router is dependent on your server that is broken. For home/DIY use, this may not be a 'problem' because you can always mess with the configuration but for anything semi-professional (if your business/job/family depends on it) you'll quickly wish you didn't. You can get a high-performance, Atom-based gigabit router with OPNsense (or pfSense) sub-$250.

For the rest, SuperMicro all the way, as long as your RAM has the right specs I've never had an issue with things that weren't "approved" as long as you're not buying from Alibaba.com - buy from good brands. In my cases, a $10 difference per part can end up making a $500 difference in the end, YMMV, any decent vendor should allow you to return it free of charge. Depending on how long you plan on keeping your hardware, 32GB is already at the low-end, I'd look for something that can scale to 128GB at least, RAM is (relatively) cheap.

Also, core frequencies are generally less important than the number of cores, bus speeds, cache sizes, memory channels and features like AES (obviously depending on your workload), most Intel processors can boost their single-core performance when necessary.

 

[joeschmuck]

I'd advocate such a solution. I believe @joeschmuck runs this it all in one too. But we run ESXi as hypervisor.

I use to run everything on one machine but have since split things up. I found that having a firewall on the same machine as other VMs was just flirting with death, death by my family when I reboot the entire system as the internet goes away for an undisclosed period of time. So I now run ESXi and VMs for Sophos UTM (firewall) and FreeNAS on my old server, this FreeNAS is for backups of my main FreeNAS, the important stuff I want backed up daily. I rarely mess with this system and it's been rock solid. My main server runs ESXi as well and many VMs, FreeNAS, Ubuntu, Windoze, etc... This FreeNAS is my main storage.

So while you could run everything from a single server, you must also realize the pitfalls of doing this. It may seem like it makes more sense to use one server but if it ends up being rebooted periodically then it may be more trouble thatn it's worth.

 

[Dice]

I agree. Thanks to @joeschmuck and @Evi Vanoost for the reminder of sound arguments - don't go overboard with complexity.

Perhaps it was a poor choice of words that I <advocate> the all-in-one box. I wanted to address the old-school safety argument where physical separated boxes sort of implied improved security. An era that virtualization has effectively commenced killing in ever increasing pace. Aspects regarding "family-proof" setups ....were not included.

Personally, I use 2 servers. Both run ESXi. FreeNAS and some VM's on one, pfsense and some VM's on the other.
I'm working towards having a short cut "redundancy" setup where if one machine needs maintenance, pfsense and other critical VM's can be migrated to the other host.
At the moment, I've not fulfilled the prophecy of a ALL-in-one, albeit the only thing remaining is a little bit of configuration coherency and ESXi work.

isnt this right what it is all about in a home lab? :)

 

[freakinlaser]

Thanks for all the advice, everyone.

I am going to stay away from running pfSense on this machine for now.
All parts have arrived with (annoyingly) the exception of the mainboard, which is still in the wind.

I decided to stick to a micro-ATX build ; this still gives me the option to replace the CPU by a 10- or 14-core model later on, and if I do run short on RAM I'll still have the option to replace it with LRDIMM .

 

[joeschmuck]

I'm working towards having a short cut "redundancy" setup where if one machine needs maintenance, pfsense and other critical VM's can be migrated to the other host.

Something else I have setup is a router which is all setup to be plugged in to replace my Sophos UTM should something go wrong with it while I'm not home, or I need to do maintenance on that machine for some period of time and I still need internet access. Someone in the family could go to the basement and easily power off the ESXi box and then power on the router. You got to think about these types of issues happening while "tech support" is away.

I guess you could establish a true failover setup but it sounds a bit complex for a home system.

I am going to stay away from running pfSense on this machine for now.

But if someone did want to run everything out of a single computer, you can do that with ESXi fairly easily, you just need to understand the pitfalls.