BUILD ESXi FreeNAS Build, Request Input

[destrekor]

To start, I know that virtualizing FreeNAS is frowned upon, but I am trying to go about it as best as possible. If anyone is willing to provide a little guidance, that would be awesome!

So I'm considering two options right now, at least in regards to buying everything brand new.

Option 1:
CPU: Xeon E3-1230 v5 w/C236 PCH
MB: likely an X11SSi-LN4F

Option 2:
Xeon D-1528 via X10SDV-6C-TLN4F

Of course the 2nd wave motherboards are slowly being revealed, and I hope Supermicro offers a better option for the D-1528, as they offer plenty of other configurations for the D1520/1521 and D1540/1541. Time will tell, and I have some time before I set aside enough money for this project.

I'll have to see what Asrock Rack and Gigabyte come up with, but I have not been impressed in the least with the configurations launched thus far, and from what I gather, I think Supermicro would be a better bet in the long run.

As for the common hardware choices thus far:
2x16GB DDR4 ECC
DOM or USB for ESXi
SSD for VMs
LSI HBA (LSI brand or a model like the IBM m1015)
8x3TB HDD
if I get a board with less than 4x RJ45 ports of any combination, then I'll add a NIC expansion card with however many I need to get to 4 at minimum. Ideally that requirement will be included on board to keep power requirements down.

I'm mainly stuck on deciding between the CPU/MB combo or the Xeon D-1528 package, as I'd like to keep power usage as low as possible, but I don't want to hamper performance to the point that the CPU is stuck drawing more power for longer to complete the same task. However, I suspect for either platform choice, the only thing that will really end up drawing energy usage out for a longer period is Plex on-the-fly transcoding. But if one can get the job done with less CPU resources and thus keep the power draw lower, that's of course a plus.

I've looked into some used servers with older generation Intel hardware, but most good deals are with Westmere chips, and I can't help but believe that choosing a Westmere CPU would end up raising power draw more often than not. If anyone can demonstrate that belief is unfounded, please do so, I don't want to be stubborn and pass up a terrific deal, but I won't mind paying more up front if that means long-term power use and heat output will be significantly lower. I'll eventually get a small rack, and as I'll likely be apartment bound for some time, I'm not sure where I'd put a rack just yet. Ideally, said rack will also hold my AV equipment as well as a switch and the ESXi server, so having the capability to use a universal remote will be a requirement. I have a Harmony right now but it's an IR model - I think the latest version is wifi so that could solve most concerns. But as I might not have many options, ensuring fan noise is kept to a minimum is definitely a concern. But beside just finding fan noise annoying, it also represents high temperatures, and I'd feel that most of this equipment will have a much longer life if temperatures are kept low. If anything comes with fans I'll likely replace them with ones that can meet my desires concerning airflow and audible volume.

For reference, I also plan to use Sophos XG on the ESXi box. And I'd like enough resources left over to host a few other VMs for lab/screwing around for self-education. More RAM may be desired in the future but I'll be comfortable for sure with FreeNAS, Sophos, and maybe another VM or two for now.

I'd love to get a board with an LSI HBA compatible with FreeNAS for passthrough, but it just doesn't seem like the right configuration exists.
And for the Skylake CPU approach, there's the issue of XHCI, as I think the same problems that exist in FreeNAS exist in ESXi, and I'd certainly like to be able to do all management through IPMI, including mounting installers or loading ESXi from a USB port, preferably an internal Type-A port.

 

[depasseg]

It might be helpful to provide a summary of the feedback you are looking for, or a list of questions.

 

[destrekor]

Ha, yeah I'm a wordy SOB.

As a summary, the feedback I am looking for most:
What route is going to be the best from a power consumption and performance standpoint, especially when factoring in cost both upfront and long-term. Though above all else, good system interoperability.

I was looking at a few systems with LSI hardware onboard, but for the Skylake C236 systems, it seems the LSI SAS 3008 is the standard, which is unfortunately not useful for FreeNAS from what I understand.
And that platform also introduces issues due to xHCI. I believe it wouldn't be an issue for FreeNAS as a VM as it would be hosted on an SSD, but I think it is an issue for IPMI and direct USB booting for ESXi.

I want to keep power use down to as low as possible for the system, but not at the expense of significant performance loss. If performance is really close between two comparably priced systems, like the Xeon D platform compared to the E3-1200 v5 series, specifically the E3-1230 v5, am I better off with the Xeon D? I really like the D-1528 but it seems the offerings are slim. I hope the choices open up later, as I won't be purchasing the hardware for at least a few more months. But what I ultimately hope to get out of this thread is a good direction so I know what I am aiming for, both in terms of budget and the general hardware. That way as I continue to search if I see a deal too good to pass up, I might purchase a component early before the rest of the system. And if it's something that will cause a sort of platform lock-in, I want to be sure I chose the right direction.

Does that help?

 

[Klontje]

I'm also pondering about what to buy/build and my options also are Xeon E3 or Xeon D. @depasseg is using the LSI3008 in his build and from what I've read he hasn't had any problems. But they are considered fairly new and thus long-term results aren't available yet. Attached is my comparison sheet, maybe it'll give you some new insights. Power consumption is hard to judge, couldn't find anything online about actual figures of the Xeon E3v5 or Xeon D but I've made a guess in my excelsheet based on previous generations Xeon.

Note, I want 10Gbe and 20SATA so that's what I've summed up in my excel...

 

[depasseg]

OK let's start with Power. Here is my freenas1 power consumption graph:

Average of 240 watts. Single socket, 128GB RAM, 12x7200 SAS drives and 2xSSD's. And it's very quiet.

Now if you were going to run more VM's than you have number of cores on a single processor, I'd suggest getting a second Processor.

In terms of boot strapping, you will need a local VMFS datastore that Freenas will boot from. I would suggest using an SSD. This will allow you to keep the system dataset on the root volume (i.e. the SSD), which could help with the drive spin down, as well as giving the ESX box a local SSD for cache.

Now, yes, even though the 3008 series with comparable SAS3 backplane has been working for me, I'd suggest a PCI HBA (M1015 or equivalent). There are less compatibility issues with the SAS2 stuff, and It might be easier to pass that through to the freenas vm.

 

[destrekor]

That's a ton of power, at least, compared to what I intend. It's terrific for a true server-grade system and what you get out of it, but that's obviously far more power (both electric and computation) than I'd like to have. Well I'd love as much computational capability as I can get, for a much smaller budget, and lower energy consumption is definitely ideal.

 

[depasseg]

So why do you want an esx server? Why not just buy a cheap/efficient firewall and a freenas box.

 

[destrekor]

I'd rather not have two separate boxes, and I do want a server to play around with. I won't need to run a ton of VMs, but I'd like to, aside from Sophos XG and FreeNAS, have a couple others that I play around with. Probably Windows Server for an AD playground, if you will, and who knows what else I'll want to take a stab at to learn a little. But the primary purpose is certainly the two primary VMs, and outside of perhaps Windows Server, I doubt I'd want many others running all the time, I can't imagine I'd have anything to do with all that with a modest home setup.

But two separate boxes doubles basic idle consumption, and is in general just not something I'd prefer to be honest. I already have an HTPC that primarily serves as a DVR with WMC for now, but the long-term plan is for the FreeNAS installation to take over that role, so long as the HDHomeRun DVR gains DRM support as is planned, but they are taking their sweet time with development, far beyond their initial goals. The recording engine is already compatible with NAS boxes, and I've seen mentioned that it will work with manual installation on FreeNAS. Couple that with Plex Media Server and my HTPC will have less tasks. I've thought about using ESXi with it and running the router through it, but it just isn't feasible in my opinion, as it's not a headless box (I have a dedicated GPU in it for light gaming and hardware accelerated video) and not a server-grade setup that I'd want to run 24/7. It's a basic i3 setup, can't remember the hardware but it's nothing incredibly special, just enough for local media playback. While ESXi can be tweaked to pass through the GPU, it's just not the right hardware combo for a server task. It goes into sleep mode when it's not doing anything, and I'd rather keep it that way.

So having two separate dedicated function servers would make for a total of 3 systems, and I'd like to get all this into a small rack along with the other AV hardware (receiver, BD player, etc). I'll add a small managed switch as well. That's all going to add up the power usage and I certainly want to keep that as low as I can.

 

[Klontje]

The 240 watts are for a Xeon E5 2637v3 with a tdp of 135w and 12 spinning disks (12 * 7w = 84w) + rest of system power draw (memory/SAS controller).

http://www.tomshardware.com/reviews/intel-xeon-e5-2600-v3-haswell-ep,3932-9.html

vs a Xeon E3 with a TDP of 80w, less memory and less disks. Below a link of previous gen power usage.

http://www.tomshardware.com/reviews/xeon-e3-1275-v3-haswell-cpu,3590-12.html

I know the figures aren't 100% comparable to a FreeNAS setup but it illustrates the difference in powerdraw.

But why don't you use VirtualBox on FreeNAS or in 10 bhyve? Only thing is that your VM's are coming online after FreeNAS so you shouldn't have any dependencies like authentication...

 

[Rand]

I am planning a comparable build in the near future so I might be able to provide some pointers in a short while (depending on board availability).

I am currently running FreeNas with W2k12+Serviio in VBox/Jail and its not working as expected (W2k12 gets network disconnect issues as soon as I put load on it) on an Intel Server board with an i3-43xx/16G and
run around 120W
I have an additional Box (Avoton build) for Sophos which needs some more oomph to handle my 200mbit Internet connection

So I am looking for
Supermicro X11SSH-ctf /w Xeon E-3 1245 ; 32 GB Ram; 8x 3 TB SATA HDD ; 4 x SAS SSD/HDD (same disk set)
Plan to run FreeNas, Sophos 9, Win2012, Serviio or similar on ESX; hopefully in the same power range as now (~140w)

Let me know if you want more details (planned as of now o/c) ;)

 

[destrekor]

The 240 watts are for a Xeon E5 2637v3 with a tdp of 135w and 12 spinning disks (12 * 7w = 84w) + rest of system power draw (memory/SAS controller).

http://www.tomshardware.com/reviews/intel-xeon-e5-2600-v3-haswell-ep,3932-9.html

vs a Xeon E3 with a TDP of 80w, less memory and less disks. Below a link of previous gen power usage.

http://www.tomshardware.com/reviews/xeon-e3-1275-v3-haswell-cpu,3590-12.html

I know the figures aren't 100% comparable to a FreeNAS setup but it illustrates the difference in powerdraw.

But why don't you use VirtualBox on FreeNAS or in 10 bhyve? Only thing is that your VM's are coming online after FreeNAS so you shouldn't have any dependencies like authentication...

I knew it wasn't quite comparable, considering the hardware was all much higher performing, including disks that use more energy than disks like the WD Reds. On that note, I still haven't settled on which HDDs to get, especially as I'm considering 8x3TB for now so I can fill up the vdev, and later upgrade each disk to a 4-6TB. I just cannot afford to grab 8 of those right now, and ~18TiB available in a RAIDZ2 configuration should tide me over for a long time... which will definitely make it easier to afford larger storage down the road as prices drop.

I should say I would prefer to get 8x 3TB disks, but at the same time, I haven't settled 100% on the system in general. It could be I wind up with a rackmount chassis that only has 6x 3.5" bays.
I would prefer lower power drives, and I cannot believe I'd truly take advantage of 7200rpm disks in my home setting with the server functioning mostly for media and backup storage. But I'd also like the higher reliability of the disks that have 10^15 or 10^16 URE rates.


But back to the topic at hand: I've considered bhyve, but I'd still feel much more comfortable with a Type1 hypervisor like ESXi. It also makes it a lot easier to upgrade the FreeNAS VM without having to worry about taking the internet offline, and also leaves the internet working fine in the event something happens to the FreeNAS installation. It's why I won't have the datastores on the zpool, so there isn't any cross dependencies. Network storage for any VMs that could use it, absolutely, but not the datastores.